{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": []
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "<center>\n",
                "    <p style=\"text-align:center\">\n",
                "        <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
                "        <br>\n",
                "        <a href=\"https://docs.arize.com/phoenix/\">Docs</a>\n",
                "        |\n",
                "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
                "        |\n",
                "        <a href=\"https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q\">Community</a>\n",
                "    </p>\n",
                "</center>\n",
                "<h1 align=\"center\">Evaluate RAG with LLM Evals</h1>\n",
                "\n",
                "In this tutorial we will look into building a RAG pipeline and evaluating it with Phoenix Evals.\n",
                "\n",
                "It has the the following sections:\n",
                "\n",
                "1. Understanding Retrieval Augmented Generation (RAG).\n",
                "2. Building RAG (with the help of a framework such as LlamaIndex).\n",
                "3. Evaluating RAG with Phoenix Evals."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Retrieval Augmented Generation (RAG)\n",
                "\n",
                "LLMs are trained on vast datasets, but these will not include your specific data (things like company knowledge bases and documentation). Retrieval-Augmented Generation (RAG) addresses this by dynamically incorporating your data as context during the generation process. This is done not by altering the training data of the LLMs but by allowing the model to access and utilize your data in real-time to provide more tailored and contextually relevant responses.\n",
                "\n",
                "In RAG, your data is loaded and prepared for queries. This process is called indexing. User queries act on this index, which filters your data down to the most relevant context. This context and your query then are sent to the LLM along with a prompt, and the LLM provides a response.\n",
                "\n",
                "RAG is a critical component for building applications such a chatbots or agents and you will want to know RAG techniques on how to get data into your application.\n",
                "\n",
                "<img src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/images/RAG_Pipeline.png\">"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Stages within RAG\n",
                "\n",
                "There are five key stages within RAG, which will in turn be a part of any larger RAG application.\n",
                "\n",
                "- **Loading**: This refers to getting your data from where it lives - whether it's text files, PDFs, another website, a database or an API - into your pipeline.\n",
                "- **Indexing**: This means creating a data structure that allows for querying the data. For LLMs this nearly always means creating vector embeddings, numerical representations of the meaning of your data, as well as numerous other metadata strategies to make it easy to accurately find contextually relevant data.\n",
                "- **Storing**: Once your data is indexed, you will want to store your index, along with any other metadata, to avoid the need to re-index it.\n",
                "\n",
                "- **Querying**: For any given indexing strategy there are many ways you can utilize LLMs and data structures to query, including sub-queries, multi-step queries, and hybrid strategies. \n",
                "- **Evaluation**: A critical step in any pipeline is checking how effective it is relative to other strategies, or when you make changes. Evaluation provides objective measures on how accurate, faithful, and fast your responses to queries are.\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Build a RAG system \n",
                "\n",
                "Now that we have understood the stages of RAG, let's build a pipeline. We will use [LlamaIndex](https://www.llamaindex.ai/) for RAG and [Phoenix Evals](https://docs.arize.com/phoenix/llm-evals/llm-evals) for evaluation.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 26,
            "metadata": {},
            "outputs": [],
            "source": [
                "!pip install -qq \"arize-phoenix[evals]\" \"llama-index>=0.10.3\" \"openinference-instrumentation-llama-index>=1.0.0\" \"llama-index-callbacks-arize-phoenix>=0.1.2\" \"llama-index-llms-openai\" \"openai>=1\" gcsfs nest_asyncio"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 27,
            "metadata": {},
            "outputs": [],
            "source": [
                "# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.\n",
                "# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.\n",
                "# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.\n",
                "import nest_asyncio\n",
                "\n",
                "nest_asyncio.apply()\n",
                "\n",
                "import os\n",
                "from getpass import getpass\n",
                "\n",
                "import pandas as pd\n",
                "import phoenix as px\n",
                "from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, set_global_handler\n",
                "from llama_index.core.node_parser import SimpleNodeParser\n",
                "from llama_index.llms.openai import OpenAI"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "During this tutorial, we will capture all the data we need to evaluate our RAG pipeline using Phoenix Tracing. To enable this, simply start the phoenix application and instrument LlamaIndex."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 28,
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "Existing running Phoenix instance detected! Shutting it down and starting a new instance...\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "🌍 To view the Phoenix app in your browser, visit http://localhost:6006/\n",
                        "📺 To view the Phoenix app in a notebook, run `px.active_session().view()`\n",
                        "📖 For more information on how to use Phoenix, check out https://docs.arize.com/phoenix\n"
                    ]
                },
                {
                    "data": {
                        "text/plain": [
                            "<phoenix.session.session.ThreadSession at 0x29e8678b0>"
                        ]
                    },
                    "execution_count": 28,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "px.launch_app()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 29,
            "metadata": {},
            "outputs": [],
            "source": [
                "set_global_handler(\"arize_phoenix\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "For this tutorial we will be using OpenAI for creating synthetic data as well as for evaluation. "
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 30,
            "metadata": {},
            "outputs": [],
            "source": [
                "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n",
                "    openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n",
                "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Load Data and Build an Index"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's use an [essay by Paul Graham](https://www.paulgraham.com/worked.html) to build our RAG pipeline."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 31,
            "metadata": {},
            "outputs": [],
            "source": [
                "import tempfile\n",
                "from urllib.request import urlretrieve\n",
                "\n",
                "with tempfile.NamedTemporaryFile() as tf:\n",
                "    urlretrieve(\n",
                "        \"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/main/data/paul_graham/paul_graham_essay.txt\",\n",
                "        tf.name,\n",
                "    )\n",
                "    documents = SimpleDirectoryReader(input_files=[tf.name]).load_data()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 32,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Define an LLM\n",
                "llm = OpenAI(model=\"gpt-4\")\n",
                "\n",
                "# Build index with a chunk_size of 512\n",
                "node_parser = SimpleNodeParser.from_defaults(chunk_size=512)\n",
                "nodes = node_parser.get_nodes_from_documents(documents)\n",
                "vector_index = VectorStoreIndex(nodes)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Build a QueryEngine and start querying."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 33,
            "metadata": {},
            "outputs": [],
            "source": [
                "query_engine = vector_index.as_query_engine()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 34,
            "metadata": {},
            "outputs": [],
            "source": [
                "response_vector = query_engine.query(\"What did the author do growing up?\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Check the response that you get from the query."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 35,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "'The author focused on writing short stories and programming, particularly experimenting with early versions of Fortran on an IBM 1401 computer during 9th grade.'"
                        ]
                    },
                    "execution_count": 35,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "response_vector.response"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "By default LlamaIndex retrieves two similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.\n",
                "\n",
                "Let's check the text in each of these retrieved nodes."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 36,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "'What I Worked On\\n\\nFebruary 2021\\n\\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn\\'t write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\\n\\nThe first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district\\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain\\'s lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\\n\\nThe language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\\n\\nI was puzzled by the 1401. I couldn\\'t figure out what to do with it. And in retrospect there\\'s not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn\\'t have any data stored on punched cards. The only other option was to do things that didn\\'t rely on any input, like calculate approximations of pi, but I didn\\'t know enough math to do anything interesting of that type. So I\\'m not surprised I can\\'t remember any programs I wrote, because they can\\'t have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn\\'t. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager\\'s expression made clear.\\n\\nWith microcomputers, everything changed.'"
                        ]
                    },
                    "execution_count": 36,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# First retrieved node\n",
                "response_vector.source_nodes[0].get_text()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 37,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "\"I remember taking the boys to the coast on a sunny day in 2015 and figuring out how to deal with some problem involving continuations while I watched them play in the tide pools. It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\\n\\nIn the summer of 2016 we moved to England. We wanted our kids to see what it was like living in another country, and since I was a British citizen by birth, that seemed the obvious choice. We only meant to stay for a year, but we liked it so much that we still live there. So most of Bel was written in England.\\n\\nIn the fall of 2019, Bel was finally finished. Like McCarthy's original Lisp, it's a spec rather than an implementation, although like McCarthy's Lisp it's a spec expressed as code.\\n\\nNow that I could write essays again, I wrote a bunch about topics I'd had stacked up. I kept writing essays through 2020, but I also started to think about other things I could work on. How should I choose what to do? Well, how had I chosen what to work on in the past? I wrote an essay for myself to answer that question, and I was surprised how long and messy the answer turned out to be. If this surprised me, who'd lived it, then I thought perhaps it would be interesting to other people, and encouraging to those with similarly messy lives. So I wrote a more detailed version for others to read, and this is the last sentence of it.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nNotes\\n\\n[1] My experience skipped a step in the evolution of computers: time-sharing machines with interactive OSes. I went straight from batch processing to microcomputers, which made microcomputers seem all the more exciting.\\n\\n[2] Italian words for abstract concepts can nearly always be predicted from their English cognates (except for occasional traps like polluzione). It's the everyday words that differ. So if you string together a lot of abstract concepts with a few simple verbs, you can make a little Italian go a long way.\""
                        ]
                    },
                    "execution_count": 37,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Second retrieved node\n",
                "response_vector.source_nodes[1].get_text()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Remember that we are using Phoenix Tracing to capture all the data we need to evaluate our RAG pipeline. You can view the traces in the phoenix application."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 38,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "phoenix URL http://localhost:6006/\n"
                    ]
                }
            ],
            "source": [
                "print(\"phoenix URL\", px.active_session().url)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We can access the traces by directly pulling the spans from the phoenix session."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 39,
            "metadata": {},
            "outputs": [],
            "source": [
                "spans_df = px.Client().get_spans_dataframe()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 40,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>name</th>\n",
                            "      <th>span_kind</th>\n",
                            "      <th>attributes.input.value</th>\n",
                            "      <th>attributes.retrieval.documents</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>a9624da12b296f3c</th>\n",
                            "      <td>llm</td>\n",
                            "      <td>LLM</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>NaN</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fc4b91051a10681a</th>\n",
                            "      <td>synthesize</td>\n",
                            "      <td>CHAIN</td>\n",
                            "      <td>What did the author do growing up?</td>\n",
                            "      <td>NaN</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1338a0c51b753dbb</th>\n",
                            "      <td>embedding</td>\n",
                            "      <td>EMBEDDING</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>NaN</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>951bd923259c3b0e</th>\n",
                            "      <td>retrieve</td>\n",
                            "      <td>RETRIEVER</td>\n",
                            "      <td>What did the author do growing up?</td>\n",
                            "      <td>[{'document.metadata': {'file_path': '/var/fol...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>9542c939a540620a</th>\n",
                            "      <td>query</td>\n",
                            "      <td>CHAIN</td>\n",
                            "      <td>What did the author do growing up?</td>\n",
                            "      <td>NaN</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                        name  span_kind              attributes.input.value  \\\n",
                            "context.span_id                                                               \n",
                            "a9624da12b296f3c         llm        LLM                                 NaN   \n",
                            "fc4b91051a10681a  synthesize      CHAIN  What did the author do growing up?   \n",
                            "1338a0c51b753dbb   embedding  EMBEDDING                                 NaN   \n",
                            "951bd923259c3b0e    retrieve  RETRIEVER  What did the author do growing up?   \n",
                            "9542c939a540620a       query      CHAIN  What did the author do growing up?   \n",
                            "\n",
                            "                                     attributes.retrieval.documents  \n",
                            "context.span_id                                                      \n",
                            "a9624da12b296f3c                                                NaN  \n",
                            "fc4b91051a10681a                                                NaN  \n",
                            "1338a0c51b753dbb                                                NaN  \n",
                            "951bd923259c3b0e  [{'document.metadata': {'file_path': '/var/fol...  \n",
                            "9542c939a540620a                                                NaN  "
                        ]
                    },
                    "execution_count": 40,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "spans_df[[\"name\", \"span_kind\", \"attributes.input.value\", \"attributes.retrieval.documents\"]].head()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Note that the traces have captured the documents that were retrieved by the query engine. This is nice because it means we can introspect the documents without having to keep track of them ourselves."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 41,
            "metadata": {},
            "outputs": [],
            "source": [
                "spans_with_docs_df = spans_df[spans_df[\"attributes.retrieval.documents\"].notnull()]"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 42,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>attributes.input.value</th>\n",
                            "      <th>attributes.retrieval.documents</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>951bd923259c3b0e</th>\n",
                            "      <td>What did the author do growing up?</td>\n",
                            "      <td>[{'document.metadata': {'file_path': '/var/fol...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                              attributes.input.value  \\\n",
                            "context.span_id                                        \n",
                            "951bd923259c3b0e  What did the author do growing up?   \n",
                            "\n",
                            "                                     attributes.retrieval.documents  \n",
                            "context.span_id                                                      \n",
                            "951bd923259c3b0e  [{'document.metadata': {'file_path': '/var/fol...  "
                        ]
                    },
                    "execution_count": 42,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "spans_with_docs_df[[\"attributes.input.value\", \"attributes.retrieval.documents\"]].head()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We have built a RAG pipeline and also have instrumented it using Phoenix Tracing. We now need to evaluate it's performance. We can assess our RAG system/query engine using Phoenix's LLM Evals. Let's examine how to leverage these tools to quantify the quality of our retrieval-augmented generation system."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Evaluation\n",
                "\n",
                "Evaluation should serve as the primary metric for assessing your RAG application. It determines whether the pipeline will produce accurate responses based on the data sources and range of queries.\n",
                "\n",
                "While it's beneficial to examine individual queries and responses, this approach is impractical as the volume of edge-cases and failures increases. Instead, it's more effective to establish a suite of metrics and automated evaluations. These tools can provide insights into overall system performance and can identify specific areas that may require scrutiny.\n",
                "\n",
                "In a RAG system, evaluation focuses on two critical aspects:\n",
                "\n",
                "- **Retrieval Evaluation**: To assess the accuracy and relevance of the documents that were retrieved\n",
                "- **Response Evaluation**: Measure the appropriateness of the response generated by the system when the context was provided."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Generate Question Context Pairs\n",
                "\n",
                "For the evaluation of a RAG system, it's essential to have queries that can fetch the correct context and subsequently generate an appropriate response.\n",
                "\n",
                "For this tutorial, let's use Phoenix's `llm_generate` to help us create the question-context pairs."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "First, let's create a dataframe of all the document chunks that we have indexed."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 43,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>text</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>I was puzzled by the 1401. I couldn't figure o...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>I remember vividly how impressed and envious I...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>I couldn't have put this into words when I was...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>The default language at Cornell was a Pascal-l...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                                text\n",
                            "0  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...\n",
                            "1  I was puzzled by the 1401. I couldn't figure o...\n",
                            "2  I remember vividly how impressed and envious I...\n",
                            "3  I couldn't have put this into words when I was...\n",
                            "4  The default language at Cornell was a Pascal-l..."
                        ]
                    },
                    "execution_count": 43,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Let's construct a dataframe of just the documents that are in our index\n",
                "document_chunks_df = pd.DataFrame({\"text\": [node.get_text() for node in nodes]})\n",
                "document_chunks_df.head()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now that we have the document chunks, let's prompt an LLM to generate us 3 questions per chunk. Note that you could manually solicit questions from your team or customers, but this is a quick and easy way to generate a large number of questions."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 44,
            "metadata": {},
            "outputs": [],
            "source": [
                "generate_questions_template = \"\"\"\\\n",
                "Context information is below.\n",
                "\n",
                "---------------------\n",
                "{text}\n",
                "---------------------\n",
                "\n",
                "Given the context information and not prior knowledge.\n",
                "generate only questions based on the below query.\n",
                "\n",
                "You are a Teacher/ Professor. Your task is to setup \\\n",
                "3 questions for an upcoming \\\n",
                "quiz/examination. The questions should be diverse in nature \\\n",
                "across the document. Restrict the questions to the \\\n",
                "context information provided.\"\n",
                "\n",
                "Output the questions in JSON format with the keys question_1, question_2, question_3.\n",
                "\"\"\""
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 45,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "The `model_name` field is deprecated. Use `model` instead.                 This will be removed in a future release.\n"
                    ]
                },
                {
                    "data": {
                        "application/vnd.jupyter.widget-view+json": {
                            "model_id": "1765156f2fb1420188c7d9b993e64067",
                            "version_major": 2,
                            "version_minor": 0
                        },
                        "text/plain": [
                            "llm_generate |          | 0/61 (0.0%) | ⏳ 00:00<? | ?it/s"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                }
            ],
            "source": [
                "import json\n",
                "\n",
                "from phoenix.evals import OpenAIModel, llm_generate\n",
                "\n",
                "\n",
                "def output_parser(response: str, index: int):\n",
                "    try:\n",
                "        return json.loads(response)\n",
                "    except json.JSONDecodeError as e:\n",
                "        return {\"__error__\": str(e)}\n",
                "\n",
                "\n",
                "questions_df = llm_generate(\n",
                "    dataframe=document_chunks_df,\n",
                "    template=generate_questions_template,\n",
                "    model=OpenAIModel(\n",
                "        model_name=\"gpt-3.5-turbo\",\n",
                "    ),\n",
                "    output_parser=output_parser,\n",
                "    concurrency=20,\n",
                ")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 46,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>question_1</th>\n",
                            "      <th>question_2</th>\n",
                            "      <th>question_3</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>What were the two main things the author worke...</td>\n",
                            "      <td>Describe the author's experience with programm...</td>\n",
                            "      <td>How did the author's experience with programmi...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>What was the author's experience with programm...</td>\n",
                            "      <td>Describe the author's transition from using th...</td>\n",
                            "      <td>How did the author's interest in programming d...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>What was the author's first experience with pr...</td>\n",
                            "      <td>Why did the author initially plan to study phi...</td>\n",
                            "      <td>What two specific influences led the author to...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>What novel by Heinlein inspired the individual...</td>\n",
                            "      <td>What programming language did the individual l...</td>\n",
                            "      <td>For their undergraduate thesis, what program d...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>What was the default language at Cornell and o...</td>\n",
                            "      <td>What was the author's experience with learning...</td>\n",
                            "      <td>What realization did the author come to during...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                          question_1  \\\n",
                            "0  What were the two main things the author worke...   \n",
                            "1  What was the author's experience with programm...   \n",
                            "2  What was the author's first experience with pr...   \n",
                            "3  What novel by Heinlein inspired the individual...   \n",
                            "4  What was the default language at Cornell and o...   \n",
                            "\n",
                            "                                          question_2  \\\n",
                            "0  Describe the author's experience with programm...   \n",
                            "1  Describe the author's transition from using th...   \n",
                            "2  Why did the author initially plan to study phi...   \n",
                            "3  What programming language did the individual l...   \n",
                            "4  What was the author's experience with learning...   \n",
                            "\n",
                            "                                          question_3  \n",
                            "0  How did the author's experience with programmi...  \n",
                            "1  How did the author's interest in programming d...  \n",
                            "2  What two specific influences led the author to...  \n",
                            "3  For their undergraduate thesis, what program d...  \n",
                            "4  What realization did the author come to during...  "
                        ]
                    },
                    "execution_count": 46,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "questions_df.head()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 47,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Construct a dataframe of the questions and the document chunks\n",
                "questions_with_document_chunk_df = pd.concat([questions_df, document_chunks_df], axis=1)\n",
                "questions_with_document_chunk_df = questions_with_document_chunk_df.melt(\n",
                "    id_vars=[\"text\"], value_name=\"question\"\n",
                ").drop(\"variable\", axis=1)\n",
                "# If the above step was interrupted, there might be questions missing. Let's run this to clean up the dataframe.\n",
                "questions_with_document_chunk_df = questions_with_document_chunk_df[\n",
                "    questions_with_document_chunk_df[\"question\"].notnull()\n",
                "]"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "The LLM has generated three questions per chunk. Let's take a quick look."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 48,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>text</th>\n",
                            "      <th>question</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "      <td>What were the two main things the author worke...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>I was puzzled by the 1401. I couldn't figure o...</td>\n",
                            "      <td>What was the author's experience with programm...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>I remember vividly how impressed and envious I...</td>\n",
                            "      <td>What was the author's first experience with pr...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>I couldn't have put this into words when I was...</td>\n",
                            "      <td>What novel by Heinlein inspired the individual...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>The default language at Cornell was a Pascal-l...</td>\n",
                            "      <td>What was the default language at Cornell and o...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>5</th>\n",
                            "      <td>I applied to 3 grad schools: MIT and Yale, whi...</td>\n",
                            "      <td>What realization did the author come to during...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>6</th>\n",
                            "      <td>So I looked around to see what I could salvage...</td>\n",
                            "      <td>What was the main reason the author decided to...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>7</th>\n",
                            "      <td>And indeed, it would seem very feeble work. On...</td>\n",
                            "      <td>What realization did the author have while vis...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>8</th>\n",
                            "      <td>And as an artist you could be truly independen...</td>\n",
                            "      <td>What was the author's initial perception of th...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>9</th>\n",
                            "      <td>I remember when my friend Robert Morris got ki...</td>\n",
                            "      <td>What was the topic chosen by the author for th...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                                text  \\\n",
                            "0  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...   \n",
                            "1  I was puzzled by the 1401. I couldn't figure o...   \n",
                            "2  I remember vividly how impressed and envious I...   \n",
                            "3  I couldn't have put this into words when I was...   \n",
                            "4  The default language at Cornell was a Pascal-l...   \n",
                            "5  I applied to 3 grad schools: MIT and Yale, whi...   \n",
                            "6  So I looked around to see what I could salvage...   \n",
                            "7  And indeed, it would seem very feeble work. On...   \n",
                            "8  And as an artist you could be truly independen...   \n",
                            "9  I remember when my friend Robert Morris got ki...   \n",
                            "\n",
                            "                                            question  \n",
                            "0  What were the two main things the author worke...  \n",
                            "1  What was the author's experience with programm...  \n",
                            "2  What was the author's first experience with pr...  \n",
                            "3  What novel by Heinlein inspired the individual...  \n",
                            "4  What was the default language at Cornell and o...  \n",
                            "5  What realization did the author come to during...  \n",
                            "6  What was the main reason the author decided to...  \n",
                            "7  What realization did the author have while vis...  \n",
                            "8  What was the author's initial perception of th...  \n",
                            "9  What was the topic chosen by the author for th...  "
                        ]
                    },
                    "execution_count": 48,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "questions_with_document_chunk_df.head(10)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Retrieval Evaluation\n",
                "\n",
                "We are now prepared to perform our retrieval evaluations. We will execute the queries we generated in the previous step and verify whether or not that the correct context is retrieved."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 49,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "🌍 To view the Phoenix app in your browser, visit http://localhost:6006/\n",
                        "📺 To view the Phoenix app in a notebook, run `px.active_session().view()`\n",
                        "📖 For more information on how to use Phoenix, check out https://docs.arize.com/phoenix\n"
                    ]
                },
                {
                    "data": {
                        "text/plain": [
                            "<phoenix.session.session.ThreadSession at 0x2afc63040>"
                        ]
                    },
                    "execution_count": 49,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# First things first, let's reset phoenix\n",
                "px.close_app()\n",
                "px.launch_app()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 50,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Question: What were the two main things the author worked on before college?\n",
                        "Answer: The author worked on writing and programming before college.\n",
                        "\n",
                        "Question: What was the author's experience with programming on the 1401 computer and why does he not remember any programs he wrote on it?\n",
                        "Answer: The author's experience with programming on the 1401 computer involved using an early version of Fortran where programs had to be typed on punch cards, stacked in the card reader, and then loaded into memory to run. The author found it challenging to work with the 1401 as the only input method was through punched cards, and without any data stored on punched cards, there were limited options for program execution. The author couldn't recall any specific programs written on the 1401 because the programs likely didn't achieve much due to the constraints of the system and the author's limited knowledge of math at that time.\n",
                        "\n",
                        "Question: What was the author's first experience with programming and what computer did they use?\n",
                        "Answer: The author's first experience with programming was in 9th grade using an IBM 1401 computer that their school district used for \"data processing.\" The computer was located in the basement of their junior high school, and they used an early version of Fortran to write programs on punch cards for this machine.\n",
                        "\n",
                        "Question: What novel by Heinlein inspired the individual to work on AI, featuring an intelligent computer called Mike?\n",
                        "Answer: The Moon is a Harsh Mistress\n",
                        "\n",
                        "Question: What was the default language at Cornell and other universities during the author's time there?\n",
                        "Answer: The default language at Cornell and other universities during the author's time there was a Pascal-like language called PL/I.\n",
                        "\n",
                        "Question: What realization did the author come to during their first year of grad school regarding AI?\n",
                        "Answer: The author realized that the type of AI being practiced at the time was not effective, as it focused on translating natural language into formal representations and adding them to a list of known things. The author recognized that this approach did not lead to actual understanding of natural language and that the explicit data structures representing concepts were not going to work.\n",
                        "\n",
                        "Question: What was the main reason the author decided to focus on Lisp and write a book about Lisp hacking?\n",
                        "Answer: The main reason the author decided to focus on Lisp and write a book about Lisp hacking was because he found Lisp interesting for its own sake, not just for its association with AI, even though that was the main reason people cared about it at the time.\n",
                        "\n",
                        "Question: What realization did the author have while visiting the Carnegie Institute in 1988?\n",
                        "Answer: The author realized that paintings were something that could be created to last, unlike software programs that would become obsolete over time.\n",
                        "\n",
                        "Question: What was the author's initial perception of the possibility of making art?\n",
                        "Answer: The author initially perceived the idea of making art as almost miraculous and something that seemed impossible to achieve. The concept of creating art seemed reserved for either individuals from the past or mysterious geniuses, making the idea of being able to make art themselves almost unimaginable.\n",
                        "\n",
                        "Question: What was the topic chosen by the author for their dissertation in order to graduate from grad school?\n",
                        "Answer: The author chose applications of continuations as the topic for their dissertation in order to graduate from grad school.\n",
                        "\n",
                        "Question: What fundamental subjects did the foundation at RISD require students to take, including drawing, color, and design?\n",
                        "Answer: The foundation at RISD required students to take classes in fundamental subjects like drawing, color, and design.\n",
                        "\n",
                        "Question: What strategy did the author use to answer the essay question during the written exam?\n",
                        "Answer: The author's strategy for answering the essay question during the written exam was to draw on personal experiences and insights gained from working on various projects, such as spam filters, painting, hosting dinners, and buying buildings for office spaces. This diverse range of experiences likely provided a rich source of material for crafting thoughtful and engaging essays.\n",
                        "\n",
                        "Question: Explain the difference between painting still lives and painting people, in terms of the subjects' ability to move and the painting process.\n",
                        "Answer: Painting still lives involves subjects that cannot move, allowing the artist to capture intricate details and nuances by closely observing and replicating what is in front of them. On the other hand, painting people presents a challenge as individuals can only sit for short periods and tend to move, requiring the artist to have a generic base to work from and then adjust it to match the specific person being painted.\n",
                        "\n",
                        "Question: How does the process of painting still lives differ from everyday visual perception?\n",
                        "Answer: The process of painting still lives differs from everyday visual perception in that painting still lives allows for a more detailed and focused observation of the subject, emphasizing specific visual cues and details that are often overlooked in everyday visual perception. This includes capturing nuances like the edge of an object or subtle color changes that contribute to creating a realistic representation of the subject.\n",
                        "\n",
                        "Question: What job did the author take at Interleaf, and what type of software did the company create?\n",
                        "Answer: The author took a job at Interleaf as a Lisp hacker. Interleaf was a company that made software for creating documents, similar to Microsoft Word.\n",
                        "\n",
                        "Question: What were some of the lessons the author learned during their time at Interleaf?\n",
                        "Answer: The author learned that it's better for technology companies to be run by product people than sales people, that editing code by too many people leads to bugs, that cheap office space can be detrimental if it's depressing, that corridor conversations are often more effective than planned meetings, that big bureaucratic customers can pose risks, and that there's a mismatch between conventional office hours and the best time for productive work.\n",
                        "\n",
                        "Question: What lesson did the author learn from their experience with Viaweb and Y Combinator?\n",
                        "Answer: The lesson the author learned from their experience with Viaweb and Y Combinator is that customs can continue to constrain individuals even after the original restrictions that caused them have disappeared. This is exemplified by the fact that customary VC practices and other established norms may not always adapt quickly to changing circumstances, creating advantages for those who are less influenced by traditional customs in fields undergoing rapid change.\n",
                        "\n",
                        "Question: What is the significance of having a signature style in the painting department at RISD?\n",
                        "Answer: Having a signature style in the painting department at RISD is significant because it serves as a visual identifier that distinguishes an artist's work from others. This distinctive style helps in establishing a unique artistic identity and can potentially lead to higher recognition and value for the artwork.\n",
                        "\n",
                        "Question: What was the main reason the author decided to drop out of RISD in 1993?\n",
                        "Answer: The main reason the author decided to drop out of RISD in 1993 was due to feeling disillusioned with the lack of substantial learning and teaching happening within the painting department at the Accademia.\n",
                        "\n",
                        "Question: What motivated the author to write another book on Lisp?\n",
                        "Answer: The author was motivated to write another book on Lisp due to financial concerns, as freelance Lisp hacking work was rare and the author did not want to switch to programming in another language like C++.\n",
                        "\n",
                        "Question: What was the initial startup idea that the author and Robert Morris decided to pursue, and why did it ultimately fail?\n",
                        "Answer: The initial startup idea that the author and Robert Morris decided to pursue was putting art galleries online. It ultimately failed because art galleries did not want to be online as it was not an effective way for them to sell their products.\n",
                        "\n",
                        "Question: What was the initial challenge faced by the author when trying to sell their website generation services?\n",
                        "Answer: The initial challenge faced by the author when trying to sell their website generation services was that it was difficult to find paying customers.\n",
                        "\n",
                        "Question: What was the motivation behind creating a web app for the store builder at Viaweb?\n",
                        "Answer: The motivation behind creating a web app for the store builder at Viaweb was driven by the desire to make it easy to use and inexpensive. This approach was influenced by the founders' lack of resources, leading them to offer the service at a low price point, which turned out to be a significant attraction for users and a challenge for competitors.\n",
                        "\n",
                        "Question: What was the initial deal made with Robert in exchange for legal work and business advice?\n",
                        "Answer: 10% of the company\n",
                        "\n",
                        "Question: Who were the three main individuals involved in the development of the software mentioned in the text, and what specific parts of the software did each of them work on?\n",
                        "Answer: The three main individuals involved in the development of the software were the author, Robert, and Trevor. The author worked on the editor, Robert worked on the shopping cart, and Trevor worked on the manager that kept track of orders and statistics.\n",
                        "\n",
                        "Question: What were the three main parts of the software developed for the ecommerce business mentioned in the text, and who was responsible for each part?\n",
                        "Answer: The three main parts of the software developed for the ecommerce business were the editor, the shopping cart, and the manager. The editor was written by the author, the shopping cart was written by Robert, and the manager was written by Trevor.\n",
                        "\n",
                        "Question: How did the founders of Viaweb unintentionally make their product more attractive to users?\n",
                        "Answer: The founders of Viaweb unintentionally made their product more attractive to users by setting a low price for their services without having a clear understanding of what businesses typically paid for such services. This low pricing strategy turned out to be a significant draw for customers and a competitive advantage, even though it was not a deliberate decision based on market insights.\n",
                        "\n",
                        "Question: Why did the author initially feel mystified and repelled by 'business' and think they needed a 'business person' to be in charge?\n",
                        "Answer: The author initially felt mystified and repelled by 'business' and thought they needed a 'business person' to be in charge because they were more focused on pursuing their passion for painting and lacked the energy or ambition for business endeavors.\n",
                        "\n",
                        "Question: What advice would the author have given to a startup at Y Combinator if they were growing 7x a year?\n",
                        "Answer: Stop being so stressed out, because you're doing fine. You're growing 7x a year. Just don't hire too many more people and you'll soon be profitable, and then you'll control your own destiny.\n",
                        "\n",
                        "Question: What was the author's initial reaction to Yahoo buying their company?\n",
                        "Answer: The author's initial reaction to Yahoo buying their company was that it felt like going from rags to riches.\n",
                        "\n",
                        "Question: Why did the author decide to leave their job at Yahoo?\n",
                        "Answer: The author decided to leave their job at Yahoo because they wanted to pursue their passion for painting.\n",
                        "\n",
                        "Question: How did the narrator's return to New York impact their life and painting career?\n",
                        "Answer: The narrator's return to New York allowed them to resume their old life, but now with the added wealth from their previous endeavors. This newfound wealth provided them with conveniences like easy access to transportation and dining options. The return to New York also reinvigorated their painting career, leading to experimentation with new painting techniques and approaches.\n",
                        "\n",
                        "Question: What new approach did the author experiment with in their painting process, involving using a printed canvas as an underpainting for a second still life?\n",
                        "Answer: The author experimented with a new approach in their painting process by using a printed canvas as an underpainting for a second still life.\n",
                        "\n",
                        "Question: Why did the author decide to start a new company in Cambridge despite not wanting to start another one?\n",
                        "Answer: The author decided to start a new company in Cambridge despite not wanting to start another one because the idea of building a web app for making web apps seemed like the future and he believed it needed to be embodied as a company.\n",
                        "\n",
                        "Question: Why did the author decide to name the new company 'Aspra' initially?\n",
                        "Answer: The author decided to name the new company 'Aspra' initially because the term \"application service provider\" (ASP) was a common name for the kind of company Viaweb was at that time.\n",
                        "\n",
                        "Question: What was the turning point for the author in terms of figuring out what to work on?\n",
                        "Answer: The turning point for the author in terms of figuring out what to work on was when they started publishing essays online.\n",
                        "\n",
                        "Question: What was the turning point for the author in figuring out what to work on?\n",
                        "Answer: The turning point for the author in figuring out what to work on was when they started publishing essays online.\n",
                        "\n",
                        "Question: What is the significance of Viaweb and Y Combinator in the context of the text?\n",
                        "Answer: Viaweb was significant as it provided users with a code editor to define their own page styles using Lisp expressions, which ran when the merchants' sites were generated. Y Combinator, on the other hand, taught a general lesson that customs can continue to constrain long after the original restrictions have disappeared. It highlighted the advantage of being independent-minded in fields undergoing rapid change and emphasized the importance of predicting which fields would be affected by such changes.\n",
                        "\n",
                        "Question: What inspired the idea for a big party at the author's house in October 2003?\n",
                        "Answer: The idea for a big party at the author's house in October 2003 was inspired by the clever suggestion of the author's friend Maria Daniels, who was one of the regular attendees at the Thursday night dinners hosted by the author.\n",
                        "\n",
                        "Question: What were some of the suggestions given by the author to fix issues in venture capital?\n",
                        "Answer: The author suggested making a larger number of smaller investments instead of a handful of giant ones, funding younger and more technical founders instead of MBAs, and allowing the founders to remain as CEO.\n",
                        "\n",
                        "Question: What was the realization that prompted the narrator to finally stop procrastinating about angel investing?\n",
                        "Answer: The realization that prompted the narrator to finally stop procrastinating about angel investing was the convergence of various factors, including the delay from VCs in making decisions, the desire to work with former colleagues on new projects, and the opportunity to start their own investment firm to implement their ideas.\n",
                        "\n",
                        "Question: What was the main motivation behind the founders of YC wanting to be an angel firm instead of a traditional VC firm?\n",
                        "Answer: The main motivation behind the founders of YC wanting to be an angel firm instead of a traditional VC firm was to provide more comprehensive support to founders in the beginning stages of their startups. They aimed to replicate the assistance they had received themselves, such as help with setting up the company, by not only making seed investments but also offering the kind of support that they found lacking when they were starting out.\n",
                        "\n",
                        "Question: What was the unique aspect of YC's funding model that set it apart from traditional VC firms?\n",
                        "Answer: The unique aspect of YC's funding model that set it apart from traditional VC firms was the batch model, where YC funded a group of startups all at once, twice a year, and then provided intensive support and guidance to them over a three-month period.\n",
                        "\n",
                        "Question: What was the motivation behind creating the Summer Founders Program and how did the founders initially attract applicants?\n",
                        "Answer: The motivation behind creating the Summer Founders Program was to provide undergraduates with an alternative to traditional summer jobs by encouraging them to start their own startups. The founders initially attracted applicants by posting an announcement on their site inviting undergraduates to apply for the program, which unexpectedly resulted in a large number of applications, including from recent graduates or soon-to-be graduates.\n",
                        "\n",
                        "Question: What was the initial deal offered to startups, and how did it compare to the deal that the founders themselves had taken?\n",
                        "Answer: The initial deal offered to startups was $6k per founder in return for 6% equity. This deal was considered fair because it was twice as good as the deal the founders themselves had taken, which was $10k for 10% equity.\n",
                        "\n",
                        "Question: What was the original intention behind Y Combinator (YC) according to the author?\n",
                        "Answer: The original intention behind Y Combinator (YC) according to the author was not to make it a full-time job. The author initially planned to focus on three things: hacking, writing essays, and working on YC.\n",
                        "\n",
                        "Question: What were the two main projects the author was working on during their time at YC?\n",
                        "Answer: Writing essays and working on YC.\n",
                        "\n",
                        "Question: What advice did Robert Morris offer to the author, and why did it make the author start thinking about quitting Y Combinator?\n",
                        "Answer: Robert Morris advised the author to ensure that Y Combinator isn't the last significant endeavor they undertake. This advice prompted the author to contemplate leaving Y Combinator because they realized that if they continued on their current path, Y Combinator would consume all their attention and become their life's sole focus. This realization led the author to consider stepping down and passing on the leadership of Y Combinator to someone else.\n",
                        "\n",
                        "Question: Why did the speaker realize that they were ready to hand over Y Combinator to someone else?\n",
                        "Answer: The speaker realized they were ready to hand over Y Combinator to someone else because they understood that if they continued on their current trajectory, Y Combinator would be the last significant thing they did, as it was consuming more of their attention. This realization, along with personal circumstances like their mother's health issues, led them to consider passing on the responsibility to someone else for the long-term sustainability of Y Combinator.\n",
                        "\n",
                        "Question: Why did the founders of YC decide to recruit Sam Altman to take over as president?\n",
                        "Answer: The founders of YC decided to recruit Sam Altman to take over as president because they wanted YC to last for a long time and believed that for that to happen, it couldn't be controlled by the original founders.\n",
                        "\n",
                        "Question: What was the author's initial decision after stopping working on YC, and why did they choose to do it?\n",
                        "Answer: The author's initial decision after stopping working on YC was to work on three things: hack, write essays, and work on YC. They chose to do this because they had not originally intended YC to be a full-time job and wanted to allocate their time among these three activities.\n",
                        "\n",
                        "Question: What was the original purpose of Lisp when it was first created?\n",
                        "Answer: The original purpose of Lisp when it was first created was to serve as a formal model of computation, as an alternative to the Turing machine.\n",
                        "\n",
                        "Question: Explain why McCarthy's original axiomatic approach was not feasible to define the missing components in the programming language.\n",
                        "Answer: McCarthy's original axiomatic approach was not feasible to define the missing components in the programming language because testing the interpreter by hand-simulating the execution of programs was reaching the limits of what could be done manually. Additionally, the complexity of the interpreter and the need for more advanced testing methods required running the interpreter, which was not possible at that time due to the lack of powerful enough computers.\n",
                        "\n",
                        "Question: What sort of trickery did the author engage in to make Bel work as an interpreter written in itself?\n",
                        "Answer: The author engaged in an egregious collection of hacks to make Bel work as an interpreter written in itself.\n",
                        "\n",
                        "Question: How did the author's experience with computers differ from the typical evolution of computers, and how did this impact their perception of microcomputers?\n",
                        "Answer: The author's experience with computers started with a TRS-80, which was a more affordable option compared to the Apple II, and this influenced their early programming endeavors. They wrote simple games, a rocket prediction program, and a word processor for their father. The limited memory capacity of the TRS-80 shaped their programming practices, such as writing in small increments due to memory constraints. This experience contrasted with the traditional mainframe computers that relied on punched cards for input. The introduction of microcomputers allowed for a more interactive and personal computing experience, where users could directly interact with the computer in real-time. This shift from mainframes to microcomputers made programming more accessible and immediate, leading the author to feel impressed and envious watching a friend type programs directly into a microcomputer.\n",
                        "\n",
                        "Question: Explain the significance of the evolution of computers mentioned in note [1] and how it impacted the author's experience.\n",
                        "Answer: The evolution of computers mentioned in note [1] highlights the author's journey from using a TRS-80 in the early days to later encountering microcomputers. This evolution marked a shift towards more accessible and user-friendly computing experiences. The author's transition to microcomputers allowed for greater flexibility and creativity in programming, enabling tasks like writing simple games and developing a word processor. This shift also facilitated a more engaging and interactive programming environment compared to the limitations of earlier computer systems. Ultimately, the evolution of computers played a crucial role in shaping the author's programming experiences and fostering a deeper interest in the field.\n",
                        "\n",
                        "Question: Explain how Interleaf, a company with smart people and impressive technology, was affected by Moore's Law in the 1990s.\n",
                        "Answer: Interleaf, a company with smart individuals and advanced technology, was impacted by Moore's Law in the 1990s as the exponential growth in the power of commodity processors, particularly Intel processors, led to the company being overshadowed. The rapid advancements in processor capabilities caused a shift in the industry, favoring high-end, specialized hardware and software companies like Interleaf to struggle against the increasing power and efficiency of mainstream processors.\n",
                        "\n",
                        "Question: Explain the significance of the code editor in Viaweb that allowed users to define their own page styles. How did this relate to Lisp expressions?\n",
                        "Answer: The code editor in Viaweb that allowed users to define their own page styles was significant because it enabled users to customize their websites without directly interacting with complex code. Users were actually editing Lisp expressions underneath, although they were not aware of it. This relationship to Lisp expressions meant that the customization process was based on manipulating Lisp code, which provided a powerful and flexible way for users to define their page styles effectively.\n",
                        "\n",
                        "Question: How did the evolution of startups impact the customs and practices of venture capitalists?\n",
                        "Answer: The evolution of startups, becoming cheaper and more common to start, led to a misalignment between the traditional customs and practices of venture capitalists and the changing startup landscape. This shift in the startup environment meant that customary VC practices, which were originally based on different constraints when startups were more expensive and less frequent, became outdated and less effective in the new scenario where startups were cheaper and more prevalent.\n",
                        "\n",
                        "Question: Explain the significance of the YC logo and its relation to the Viaweb logo.\n",
                        "Answer: The YC logo was designed as an inside joke in relation to the Viaweb logo. The Viaweb logo featured a white V on a red circle, so the YC logo was created with a white Y on an orange square. This choice of design for the YC logo was a playful nod to the original Viaweb logo, incorporating a similar style while giving it a unique twist.\n",
                        "\n",
                        "Question: How did leaving YC impact the relationship between the author and Jessica?\n",
                        "Answer: Leaving YC led to a change in the relationship between the author and Jessica, as the author asked Jessica if she wanted to be president but she declined. This decision prompted the author to recruit Sam Altman instead, leading to a reorganization of YC where the author and Jessica transitioned to different roles within the organization.\n",
                        "\n",
                        "Question: Describe the author's experience with programming on the IBM 1401 in 9th grade.\n",
                        "Answer: The author's experience with programming on the IBM 1401 in 9th grade was challenging and limited. They found it puzzling as they couldn't figure out what to do with it due to the constraints of the system. The only input method was through punched cards, which the author didn't have access to. Without much mathematical knowledge to perform complex calculations, the author struggled to create meaningful programs on the IBM 1401. Their most vivid memory was encountering a program that didn't terminate, highlighting the technical limitations and social implications of such errors in a non-time-sharing environment.\n",
                        "\n",
                        "Question: Describe the author's transition from using the 1401 computer to a microcomputer. How did this change his programming experience?\n",
                        "Answer: The author initially found the 1401 computer puzzling as it required data stored on punched cards for input, limiting his programming options. With the introduction of microcomputers, such as the TRS-80, the author's programming experience changed significantly. He was able to have a computer right in front of him, responding to his keystrokes in real-time, which was a stark contrast to the batch processing method of the 1401. This shift allowed him to directly type programs into the computer, enabling more interactive and immediate programming experiences compared to the punched card system.\n",
                        "\n",
                        "Question: Why did the author initially plan to study philosophy in college and what did they discover about the field once they started?\n",
                        "Answer: The author initially planned to study philosophy in college because they believed it would help them understand the nature of intelligence. However, once they started studying philosophy, they discovered that the field did not provide the practical tools they needed to achieve their goal of understanding intelligence.\n",
                        "\n",
                        "Question: What programming language did the individual learn in order to teach themselves AI, as it was regarded as the language of AI in the mid-1980s?\n",
                        "Answer: The individual learned Lisp in order to teach themselves AI, as it was regarded as the language of AI in the mid-1980s.\n",
                        "\n",
                        "Question: What was the author's experience with learning Lisp and how did it impact their understanding of programming?\n",
                        "Answer: The author's experience with learning Lisp was transformative. Learning Lisp expanded their concept of a program rapidly, pushing the boundaries of their understanding for years. It was a significant departure from the default Pascal-like language they were accustomed to, and it opened up new possibilities in programming that they had not previously encountered.\n",
                        "\n",
                        "Question: Why did the author decide to focus on Lisp after realizing the limitations of AI as practiced at the time?\n",
                        "Answer: The author decided to focus on Lisp after realizing the limitations of AI as practiced at the time because they found Lisp interesting for its own sake, not just for its association with AI. They believed that Lisp had value beyond its AI applications and decided to delve into it further, eventually leading to the decision to write a book about Lisp hacking.\n",
                        "\n",
                        "Question: How does the author describe the uneasy alliance between theory and systems in computer science?\n",
                        "Answer: The author describes the uneasy alliance between theory and systems in computer science as a division where theory people focus on proving things, while systems people concentrate on building things. The author expresses a preference for building things, acknowledging the importance of theory but finding the act of building more exciting.\n",
                        "\n",
                        "Question: Why did the author decide to start taking art classes at Harvard?\n",
                        "Answer: The author decided to start taking art classes at Harvard because they realized that creating art was something that could last and be a viable means of living independently.\n",
                        "\n",
                        "Question: How did the author manage to graduate from the PhD program in computer science despite being focused on art and other projects?\n",
                        "Answer: The author managed to graduate from the PhD program in computer science by deciding to write a dissertation in a short period of time, reusing parts of their existing work. This quick decision allowed them to respond positively when asked about their progress towards graduation, even though they had not written a word of their dissertation at that point.\n",
                        "\n",
                        "Question: Which two art schools did the author apply to and which one accepted them?\n",
                        "Answer: The author applied to RISD in the US and the Accademia di Belli Arti in Florence. RISD accepted the author.\n",
                        "\n",
                        "Question: Why did the Accademia require stranieri (foreigners) to take an entrance exam, and what was the potential purpose behind this requirement?\n",
                        "Answer: The Accademia required stranieri (foreigners) to take an entrance exam as a way to potentially exclude them. This requirement may have been implemented to prevent the Italian students from being outnumbered by the large number of foreigners attracted to studying art in Florence.\n",
                        "\n",
                        "Question: Describe the dynamics between the students and faculty in the painting department at the Accademia as mentioned in the text.\n",
                        "Answer: The students and faculty in the painting department at the Accademia had an arrangement where the students did not require the faculty to teach anything, and in return, the faculty did not require the students to learn anything. Despite outwardly adhering to the conventions of a 19th-century atelier, the focus seemed to be more on socializing and imitating things from American art magazines rather than actively engaging in painting.\n",
                        "\n",
                        "Question: How does the author emphasize visual cues in still life paintings to make them more realistic than photographs?\n",
                        "Answer: The author emphasizes visual cues in still life paintings by subtly highlighting details such as the abrupt color changes at object edges. This approach aims to create paintings that are more realistic than photographs not just metaphorically, but in a strict information-theoretic sense.\n",
                        "\n",
                        "Question: What was the author's experience like studying at the Accademia, and why did they ultimately decide to return to the US?\n",
                        "Answer: The author's experience at the Accademia involved a lack of substantial learning as the faculty and students had an arrangement where teaching and learning were not emphasized. The author received good grades for working hard but felt that the institution wasn't teaching much besides Italian. Ultimately, the author decided to return to the US because their money was running out, and they needed to find a job to sustain themselves financially.\n",
                        "\n",
                        "Question: How did the author describe their experience working at Interleaf, and what was the main reason for their irresponsibility?\n",
                        "Answer: The author described their experience working at Interleaf as not very positive. They mentioned that the company had added a scripting language inspired by Emacs, making it a dialect of Lisp, but the author found it challenging as they did not know C and did not want to learn it. The main reason for their irresponsibility was their discomfort with the traditional concept of a programming job that required showing up every day during specific working hours, which they found unnatural.\n",
                        "\n",
                        "Question: How did the author manage to save enough money to go back to RISD and pay off their college loans while working at Interleaf?\n",
                        "Answer: The author managed to save enough money to go back to RISD and pay off their college loans while working at Interleaf by getting paid significant amounts of money, especially compared to what they were used to as an art student. Despite living frugally, the author was earning more than four times their previous daily budget even when just sitting in meetings. This increase in income allowed them to save up and cover their expenses, enabling them to return to RISD and pay off their college loans.\n",
                        "\n",
                        "Question: How did the author survive financially while attending RISD?\n",
                        "Answer: The author survived financially while attending RISD by doing freelance work for a group that handled projects for clients.\n",
                        "\n",
                        "Question: How did the author's experience at RISD differ from the students who were earnest and focused on improving their drawing skills?\n",
                        "Answer: The author's experience at RISD differed from the earnest students who were focused on improving their drawing skills in that the author was essentially teaching themselves to paint and did not feel the need to conform to the idea of developing a distinctive signature style. While the earnest students were dedicated to enhancing their drawing abilities and were somewhat disheartened by the environment at RISD, the author was more aligned with those who were not seeking a signature style but rather learning to paint independently.\n",
                        "\n",
                        "Question: How did the author end up living in Yorkville, New York, in 1993?\n",
                        "Answer: The author ended up living in Yorkville, New York, in 1993 because a rent-controlled apartment in a building owned by a friend's mother was becoming vacant, and the author was offered the opportunity to move in.\n",
                        "\n",
                        "Question: Who was Idelle Weber and what role did she play in the author's life?\n",
                        "Answer: Idelle Weber was a painter, specifically one of the early photorealists. In the author's life, she was a beloved teacher whom the author had taken a painting class with at Harvard. After the author moved to New York, he became her de facto studio assistant.\n",
                        "\n",
                        "Question: How did the author's perception of the World Wide Web influence their decision to pivot from art galleries to online stores?\n",
                        "Answer: The author's perception of the World Wide Web as a significant development, similar to the impact of graphical user interfaces on microcomputers, led them to recognize the potential of online stores as a lucrative opportunity. This realization prompted the shift from focusing on art galleries to developing software for online stores, as they observed the similarities between the websites they were generating for galleries and the emerging concept of internet storefronts.\n",
                        "\n",
                        "Question: Describe the moment when the author had a breakthrough idea while staying at Robert's apartment. What was the idea and how did it change their approach to building online stores?\n",
                        "Answer: The author had a breakthrough idea when they realized the potential of the World Wide Web after being introduced to it by Robert Morris. The idea was to shift their focus from putting art galleries online to building online stores. This change in direction allowed them to leverage their existing knowledge and skills to create software for online stores, as they noticed the similarities between the websites they were generating for galleries and the emerging online stores. This shift in focus enabled them to develop a WYSIWYG site builder, emphasizing the importance of high production values to make users appear legitimate.\n",
                        "\n",
                        "Question: How did the founders of Viaweb secure seed funding and what was the arrangement with the investor?\n",
                        "Answer: The founders of Viaweb secured seed funding by approaching successful startup founders, who not only provided the funding but also served as a source of advice. The arrangement with the investor was such that the founders of Viaweb would receive funding and guidance from these successful startup founders, leveraging their experience and knowledge in addition to the financial support.\n",
                        "\n",
                        "Question: How did the author's negative net worth impact the need for seed funding?\n",
                        "Answer: The author's negative net worth necessitated the need for seed funding to sustain living expenses, as the amount in the bank was outweighed by the tax debt owed to the government.\n",
                        "\n",
                        "Question: What was the initial concern regarding the timing of opening the online stores in January 1996, and why was it actually advantageous to wait a few months?\n",
                        "Answer: The initial concern regarding the timing of opening the online stores in January 1996 was that the founders worried they were late to the market due to the talk about ecommerce in the press. However, it turned out to be advantageous to wait a few months because despite the early concerns, not many people actually wanted online stores at that time.\n",
                        "\n",
                        "Question: Why was the pricing of the ecommerce software developed by the startup considered a big attraction, and what was the pricing strategy employed by the company?\n",
                        "Answer: The pricing of the ecommerce software developed by the startup was considered a big attraction because it was made more inexpensive than competitors realized. The company charged $100 a month for a small store and $300 a month for a big one. This low pricing strategy was not based on a clever insight but rather on the fact that the founders had no idea what businesses typically paid for such services, and $300 a month seemed like a significant amount to them.\n",
                        "\n",
                        "Question: What was one of the key lessons the founders learned about retail while building stores for users?\n",
                        "Answer: One of the key lessons the founders learned about retail while building stores for users was the importance of having a closeup image of a product, such as a man's shirt collar, rather than a picture of the whole shirt when only small images were available.\n",
                        "\n",
                        "Question: What did the author realize was the ultimate test of a startup, and how did their company's growth rate compare to this test?\n",
                        "Answer: The author realized that the growth rate is the ultimate test of a startup. Their company's growth rate was significant, growing from about 70 stores at the end of 1996 to about 500 at the end of 1997, which indicated a growth rate of 7x a year. This growth rate was considered positive and indicated that the company was on the right track towards profitability and controlling its own destiny.\n",
                        "\n",
                        "Question: Why did the author hire more people for the company, even though it delayed reaching breakeven?\n",
                        "Answer: The author hired more people for the company because the investors wanted it and because during the Internet Bubble era, it was common for startups to expand their workforce significantly. This expansion was driven by the perception that a company with only a few employees might appear inexperienced or unprofessional in the industry at that time.\n",
                        "\n",
                        "Question: Why did the author feel worn out during the summer of 1998 to the summer of 1999?\n",
                        "Answer: The author felt worn out during the summer of 1998 to the summer of 1999 due to the effort and stress of running Viaweb, combined with fatigue and the challenging work environment at Yahoo in Santa Clara.\n",
                        "\n",
                        "Question: What was the initial reaction of the author's boss at Yahoo when they announced their plans to leave?\n",
                        "Answer: The initial reaction of the author's boss at Yahoo when they announced their plans to leave was a long conversation about the author's plans.\n",
                        "\n",
                        "Question: What new approach did the narrator experiment with in their still life paintings?\n",
                        "Answer: The narrator experimented with emphasizing visual cues in their still life paintings to make them more realistic than photographs.\n",
                        "\n",
                        "Question: What realization did the author come to while looking for an apartment to buy in New York, in comparison to their experiences in Cambridge?\n",
                        "Answer: The author realized that there wasn't a neighborhood in New York similar to Cambridge, after considering various options and even visiting the actual Cambridge.\n",
                        "\n",
                        "Question: What was the initial reaction of Robert when approached to work on the new idea, and why did he refuse to work on it?\n",
                        "Answer: Robert initially expressed skepticism when approached to work on the new idea, but ultimately refused to work on it due to his commitments as a postdoc at MIT and the significant time investment required for the project.\n",
                        "\n",
                        "Question: What realization led the author to switch from building a company to working on an open source project?\n",
                        "Answer: Realizing that running a big company was not something the author wanted to do anymore, especially since the initial motivation for starting a company was no longer relevant, prompted the author to switch from building a company to working on an open source project.\n",
                        "\n",
                        "Question: What surprised the author about publishing content on the web compared to the print era?\n",
                        "Answer: What surprised the author about publishing content on the web compared to the print era was the realization that anyone could publish anything and reach an audience without the traditional barriers imposed by editors and limited publishing channels.\n",
                        "\n",
                        "Question: Why did the author find encouragement in the marginal nature of online essays at first?\n",
                        "Answer: The author found encouragement in the marginal nature of online essays at first because they believed that despite initially being perceived as less prestigious and more like rants, this marginal medium allowed for the publication of essays that had never been written before due to the limitations of the print era.\n",
                        "\n",
                        "Question: How does the author's experience with writing essays and painting relate to the concept of prestige in work?\n",
                        "Answer: The author's experience with writing essays and painting demonstrates a pattern of being drawn to work that is considered unprestigious. Despite the lack of prestige associated with still life painting, starting Viaweb and Y Combinator, and writing essays online, the author found value in pursuing these endeavors. This suggests that the author values authenticity and genuine interest over seeking prestige or impressing others in their choice of work.\n",
                        "\n",
                        "Question: How did Jessica Livingston's perception of startups change after meeting friends of the author from the startup world?\n",
                        "Answer: Jessica Livingston's perception of startups changed after meeting friends of the author from the startup world by realizing how different reality was compared to what her Boston investment bank thought, and how colorful the stories of startup founders were.\n",
                        "\n",
                        "Question: Why did the author decide to start angel investing after giving a talk at the Harvard Computer Society?\n",
                        "Answer: The author decided to start angel investing after giving a talk at the Harvard Computer Society because they realized that successful startup founders were the best sources of seed funding and valuable advice.\n",
                        "\n",
                        "Question: How did the narrator's discussion with Robert and Trevor lead to the idea of starting their own investment firm?\n",
                        "Answer: The narrator's discussion with Robert and Trevor led to the idea of starting their own investment firm because they were scheming about projects they could collaborate on and missed working together. As they were walking home from dinner, the idea to start their own investment firm emerged as a way to implement the ideas they had been discussing. The narrator decided to fund the firm, Jessica would quit her job to work for it, and Robert and Trevor would join as partners.\n",
                        "\n",
                        "Question: How did YC differ from traditional VC firms in terms of funding and organization?\n",
                        "Answer: YC differed from traditional VC firms in terms of funding and organization by focusing on making seed investments to help startups in the beginning, rather than solely making big, million dollar investments like VC firms. Additionally, YC was not organized as a fund and was funded with the founders' own money, which was a departure from the typical structure of professional VC firms.\n",
                        "\n",
                        "Question: How did the idea for the Summer Founders Program come about, and what was the initial purpose behind it?\n",
                        "Answer: The idea for the Summer Founders Program came about when the individuals behind it wanted to gain experience as investors by funding a group of startups all at once. The initial purpose behind the program was to provide undergraduates with an opportunity to start their own startups during the summer instead of taking temporary jobs at tech companies, allowing them to have a more interesting and entrepreneurial experience.\n",
                        "\n",
                        "Question: Describe the selection process for the Summer Founders Program and highlight some of the successful individuals who were part of the first batch of funded startups.\n",
                        "Answer: The selection process for the Summer Founders Program involved receiving 225 applications, inviting about 20 groups for in-person interviews, and ultimately selecting 8 groups to fund. The individuals in the first batch of funded startups included notable names such as the founders of reddit, Justin Kan and Emmett Shear who later founded Twitch, Aaron Swartz who contributed to writing the RSS spec, and Sam Altman who became the second president of Y Combinator.\n",
                        "\n",
                        "Question: How did funding startups in batches help address a common problem faced by founders?\n",
                        "Answer: Funding startups in batches helped address the common problem faced by founders of isolation. Being part of a batch provided founders with colleagues who understood the challenges they were encountering and could offer insights on how to overcome them.\n",
                        "\n",
                        "Question: Why did the author change the name and topic of the news aggregator from Startup News to Hacker News?\n",
                        "Answer: The author changed the name and topic of the news aggregator from Startup News to Hacker News because they got tired of reading only about startups and realized that the target audience they wanted to reach was not just current startup founders, but also future startup founders.\n",
                        "\n",
                        "Question: What aspects of the job at YC did the author find engaging, despite there being parts they didn't like?\n",
                        "Answer: The author found the engagement in the job at YC through the problems that came to them from startups in the program. They mentioned that the problems were varied and that working with effective founders was engaging. Despite not liking aspects such as disputes between cofounders, dealing with dishonesty, and conflicts with those mistreating startups, the author worked hard even on these parts of the job.\n",
                        "\n",
                        "Question: What were some of the challenges the author faced while working at Y Combinator, and how did they approach these challenges?\n",
                        "Answer: The author faced challenges such as disputes between cofounders, identifying dishonesty, and dealing with individuals mistreating startups while working at Y Combinator. Despite not enjoying these aspects of the job, the author worked diligently on them. The author was motivated by the belief that as the leader, they needed to work the hardest to set an example for others. This dedication to overcoming challenges and working hard, even on tasks they disliked, was a key approach the author took while working at Y Combinator.\n",
                        "\n",
                        "Question: What was the main reason for the decision to recruit Sam Altman as the new president of Y Combinator?\n",
                        "Answer: The main reason for the decision to recruit Sam Altman as the new president of Y Combinator was the founders' desire for YC to last for a long time and to ensure that it couldn't be controlled by the original founders.\n",
                        "\n",
                        "Question: What was the initial reason Sam Altman gave for not wanting to be president of YC?\n",
                        "Answer: Sam Altman initially declined the offer to be president of Y Combinator because he wanted to start a startup focused on making nuclear reactors.\n",
                        "\n",
                        "Question: Describe the author's experience with painting in 2014 and the reason behind why they stopped painting in November.\n",
                        "Answer: The author's experience with painting in 2014 involved feeling a lack of energy and ambition, partly due to not knowing many people in California and living in a remote location in the Santa Cruz Mountains. Despite trying to paint, the author struggled with motivation. In November, the author stopped painting and returned to New York out of desperation, where they found a sense of familiarity and resumed their old life. The author's return to New York brought about a change in their financial situation, leading to a period of excitement and experimentation with painting techniques, such as creating still lifes using a new method involving photography and printing on canvas.\n",
                        "\n",
                        "Question: How did Lisp evolve from being a formal model of computation to a programming language?\n",
                        "Answer: Lisp evolved from being a formal model of computation to a programming language when John McCarthy's grad student, Steve Russell, suggested translating McCarthy's interpreter into IBM 704 machine language. This step allowed Lisp to start being used as a programming language in the traditional sense, expanding its capabilities beyond just interpreting Lisp expressions.\n",
                        "\n",
                        "Question: What challenges did the author face in testing the interpreter for the programming language, and how were these challenges overcome?\n",
                        "Answer: The author faced challenges in testing the interpreter for the programming language due to limitations in the power of computers at the time. The interpreter had to be hand-simulated for testing, which was becoming impractical as the complexity increased. To overcome this challenge, the author had to wait until computers became powerful enough to run the interpreter directly, allowing for more comprehensive testing and development of the programming language.\n",
                        "\n",
                        "Question: How did the author manage to keep track of the code and handle errors while working on an interpreter written in itself?\n",
                        "Answer: The author found it challenging to keep track of the code and handle errors while working on an interpreter written in itself due to the complexity of the problem. This complexity made it difficult to understand what was happening at different levels, and errors could become nearly indecipherable by the time they were encountered.\n",
                        "\n",
                        "Question: Why did the author and their family move to England in 2016, and what was the initial plan versus the eventual outcome of their stay?\n",
                        "Answer: The author and their family moved to England in 2016 to allow their kids to experience living in another country. Initially, they intended to stay for a year, but they ended up liking it so much that they continued to live there.\n",
                        "\n",
                        "Question: Describe the author's daily walk from Piazza San Felice 4 to the Accademia in Florence as detailed in note [3]. What observations did the author make during this walk?\n",
                        "Answer: The author's daily walk from Piazza San Felice 4 to the Accademia in Florence involved passing by several significant landmarks. The walk went straight down the spine of old Florence, past the Pitti Palace, across a bridge, past Orsanmichele, between the Duomo and the Baptistery, and then up Via Ricasoli to Piazza San Marco. During this walk, the author observed Florence at street level in various conditions, from empty dark winter evenings to crowded summer days filled with tourists.\n",
                        "\n",
                        "Question: What is the significance of launching a software privately before launching it publicly, especially in the case of an online store builder?\n",
                        "Answer: Launching a software privately before launching it publicly, especially in the case of an online store builder, allows for testing and refining the product in a controlled environment. It enables the creators to gather feedback, identify and fix any issues, and ensure that the software functions as intended before making it available to a wider audience. This approach helps in improving the user experience, addressing any technical glitches, and increasing the chances of a successful public launch.\n",
                        "\n",
                        "Question: What was the author's experience when reading comments about Lisp being better than other languages? How did they respond to the criticism?\n",
                        "Answer: The author's experience when reading comments about Lisp being better than other languages was that the comments were full of angry people questioning how they could claim Lisp was superior. In response to the criticism, the author acknowledged that some people may feel sorry for them, but they stated that facing such criticism has always been a part of their experience, as essays often need to inform readers of new perspectives, which can sometimes be met with resistance.\n",
                        "\n",
                        "Question: What was the reasoning behind the choice of the name 'Y Combinator' and the color orange for the brand?\n",
                        "Answer: The reasoning behind the choice of the name 'Y Combinator' was to avoid having a regional name and to reflect a cool trick in the lambda calculus. The color orange was chosen because it was warm and distinctive, unlike the traditional colors used by other venture capitalists at the time.\n",
                        "\n",
                        "Question: Discuss the concept of 'deal flow' and how YC aims to challenge this notion.\n",
                        "Answer: The term \"deal flow\" typically refers to the number of new startups available for investment at any given time. YC challenges this notion by actively creating opportunities for startups that may not have otherwise existed. By running programs like the Summer Founders Program and funding a batch of startups all at once, YC increases the pool of potential investments and founders, thereby expanding and diversifying the traditional concept of deal flow. This approach allows YC to play a proactive role in shaping the startup ecosystem and fostering innovation beyond what would naturally occur in the market.\n",
                        "\n",
                        "Question: What is the significance of discussing space aliens in relation to the concept of invented vs discovered?\n",
                        "Answer: Discussing space aliens in relation to the concept of invented vs discovered helps to illustrate that certain mathematical and scientific concepts, such as the Pythagorean theorem and programming languages like Lisp, are likely to be universal and known by advanced civilizations. This discussion suggests that there may be a commonality in knowledge across different intelligent beings, implying that certain discoveries in mathematics and technology could be inherent and not solely human inventions.\n",
                        "\n",
                        "Question: How did the author's experience with programming change with the introduction of microcomputers?\n",
                        "Answer: With the introduction of microcomputers, the author's experience with programming changed significantly. They were able to have a computer right in front of them on a desk, allowing for immediate response to keystrokes while the program was running. This was a stark contrast to the previous method of using punched cards for input and waiting for the program to finish processing. The availability of microcomputers enabled the author to start programming more actively, creating simple games, predictive programs, and even a word processor that their father used for writing.\n",
                        "\n",
                        "Question: How did the author's interest in programming develop over time, from watching a friend build a microcomputer to eventually convincing his father to buy one for him? How did he use the TRS-80 computer for programming projects?\n",
                        "Answer: The author's interest in programming developed from being impressed and envious watching a friend build a microcomputer to eventually convincing his father to buy a TRS-80 computer in about 1980. With the TRS-80, the author started programming by writing simple games, creating a program to predict the height of model rockets, and developing a word processor that his father used to write at least one book. Despite the limited memory capacity allowing only about 2 pages of text to be stored at a time, the word processor was an improvement over a typewriter.\n",
                        "\n",
                        "Question: What two specific influences led the author to switch their focus to AI in the mid 1980s?\n",
                        "Answer: A novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU.\n",
                        "\n",
                        "Question: For their undergraduate thesis, what program did the individual reverse-engineer, expressing excitement and belief that it was climbing the lower slopes of intelligence?\n",
                        "Answer: For their undergraduate thesis, the individual reverse-engineered the program SHRDLU, expressing excitement and belief that it was climbing the lower slopes of intelligence.\n",
                        "\n",
                        "Question: What realization did the author come to during their first year of grad school regarding the practice of AI at the time?\n",
                        "Answer: The author realized that the type of AI being practiced at the time, which involved programs translating natural language into formal representations, was not effective in truly understanding natural language. The author recognized that these programs only demonstrated a subset of natural language that was a formal language, but not a comprehensive understanding of language.\n",
                        "\n",
                        "Question: How does the author describe the relationship between theory and systems in Computer Science, and which aspect did they prefer to focus on?\n",
                        "Answer: The author describes the relationship between theory and systems in Computer Science as an uneasy alliance. Theory involves proving things, while systems involve building things. The author preferred to focus on building things rather than on theory.\n",
                        "\n",
                        "Question: What realization did the author have while visiting the Carnegie Institute in 1988?\n",
                        "Answer: The author realized that paintings were something that could be created to last, unlike software programs that would become obsolete over time.\n",
                        "\n",
                        "Question: How did the author view the people who made art before deciding to pursue it themselves?\n",
                        "Answer: The author viewed the people who made art as either belonging to a different species or being mysterious geniuses who lived long ago, doing strange things that were featured in profiles in Life magazine. The idea of being able to make art themselves seemed almost miraculous to the author before they decided to pursue it.\n",
                        "\n",
                        "Question: If the author could rewrite their dissertation, what topic would they choose instead of applications of continuations?\n",
                        "Answer: The author would choose to write about macros and embedded languages instead of applications of continuations.\n",
                        "\n",
                        "Question: What was the big surprise the author received towards the end of the summer and how did it impact their plans?\n",
                        "Answer: The big surprise the author received towards the end of the summer was the realization that their boss at Yahoo thought they were lying about quitting to paint and assumed they were leaving to start a new startup. This impacted their plans as it made them reflect on the perception others had of their decision to leave a lucrative job to pursue painting, leading to a sense of misunderstanding and potential consequences in the tech industry during the Internet Bubble period.\n",
                        "\n",
                        "Question: Describe the arrangement between the students and faculty in the painting department at the Accademia, as mentioned in the context information.\n",
                        "Answer: The students and faculty in the painting department at the Accademia had reached an agreement where the faculty did not feel obligated to teach the students, and in return, the students were not required to learn anything. This arrangement allowed for a lack of academic rigor, with little emphasis on instruction or learning. Additionally, the students and faculty outwardly adhered to the traditional practices of a 19th-century atelier, despite the lack of substantial teaching and learning happening within the department.\n",
                        "\n",
                        "Question: How did the author's painting style differ when painting still lives compared to painting people, according to the context information?\n",
                        "Answer: When painting still lives, the author's approach involved capturing details pixel by pixel from what was being seen, allowing for a more meticulous reproduction of the subject. In contrast, when painting people, the author typically started with a generic person and then made modifications to match the specific individual being painted.\n",
                        "\n",
                        "Question: Why does the author prefer painting still lives over painting people, and what aspect of visual perception does he find intriguing in the process?\n",
                        "Answer: The author prefers painting still lives over painting people because the subject of still lives does not move, unlike people who can't sit very still for long periods. In painting still lives, the author can copy pixel by pixel from what is being seen, allowing for a more detailed and accurate representation. The author finds the aspect of visual perception intriguing where everyday visual perception is handled by low-level processes that do not provide detailed information like the shape and position of every leaf on a bush, as it would be distracting in everyday life to notice such details.\n",
                        "\n",
                        "Question: What did the author learn about software development while working at Interleaf, and how did this experience shape their perspective on low end software versus high end software?\n",
                        "Answer: The author learned several things about software development while working at Interleaf. They discovered that it's better for technology companies to be run by product people rather than sales people, that editing code by too many people leads to bugs, that cheap office space can be detrimental if it's depressing, that planned meetings are less effective than corridor conversations, that big bureaucratic customers can pose risks, and that there's a mismatch between conventional office hours and the optimal time for programming. Additionally, the author realized that being the \"entry level\" option is advantageous in the software industry as the low end tends to dominate the high end. This understanding led the author to believe that it's beneficial to prioritize being the entry level option, even if it means sacrificing prestige, as failing to do so could result in being overshadowed by competitors.\n",
                        "\n",
                        "Question: How did the author's financial situation change after working at Interleaf, and what was the impact on their daily budget compared to when they were in Florence?\n",
                        "Answer: After working at Interleaf, the author's financial situation improved significantly. They mentioned that in Florence, their budget for everything else besides rent was $7 a day. However, after working at Interleaf, they were getting paid more than 4 times that amount every hour, even when just sitting in a meeting. This increase in income allowed the author to save enough money to go back to RISD and pay off their college loans.\n",
                        "\n",
                        "Question: According to the author, what is the significance of being the 'entry level' option in a business?\n",
                        "Answer: Being the 'entry level' option in a business is important because it helps prevent being overshadowed by competitors and getting squeezed against a ceiling. This strategy is emphasized as a way to avoid losing out to others and to maintain a competitive edge in the market.\n",
                        "\n",
                        "Question: What is a 'signature style' in the context of painting, and how is it related to the work of artists like Roy Lichtenstein?\n",
                        "Answer: A 'signature style' in painting is a distinctive visual characteristic that immediately identifies the work as belonging to a specific artist. It is akin to a unique trademark that sets the artist's work apart from others. For instance, when a painting resembles a particular type of cartoon, viewers can recognize it as being created by Roy Lichtenstein. This distinctive style becomes a recognizable feature of the artist's work, making it easily identifiable and often sought after by collectors.\n",
                        "\n",
                        "Question: What led the author to drop out of RISD in 1993 and move to New York?\n",
                        "Answer: The author decided to drop out of RISD in 1993 and move to New York because a rent-controlled apartment in New York, owned by a friend's mother, was becoming vacant and it was not much more expensive than the author's current place. Additionally, New York was perceived as a place where artists were located, which influenced the author's decision to move.\n",
                        "\n",
                        "Question: Who was Idelle Weber and what role did she play in the author's life during his time in New York?\n",
                        "Answer: Idelle Weber was a painter, known as one of the early photorealists. During the author's time in New York, he became her de facto studio assistant.\n",
                        "\n",
                        "Question: What was the author's initial idea for getting rich and how did it change after learning about the World Wide Web?\n",
                        "Answer: The author's initial idea for getting rich was to start a company to put art galleries online. However, after learning more about the World Wide Web and its potential impact, the author realized that online stores were becoming popular and decided to shift focus towards building online stores instead.\n",
                        "\n",
                        "Question: What challenges did the author and Robert face when trying to sell their software to art galleries, and how did they eventually pivot their business idea?\n",
                        "Answer: The author and Robert faced challenges when trying to sell their software to art galleries as the galleries were not interested in being online and did not see the value in the idea. They struggled to get galleries to pay for their services, with most only agreeing to let them create sites for free. However, they eventually pivoted their business idea when they noticed the emergence of online stores that resembled the sites they were generating for galleries. Realizing they could apply their expertise to build online stores, they shifted their focus from art galleries to developing software for online stores, which led them to explore the concept of web apps and building a store builder that could be controlled through a browser without the need for client software.\n",
                        "\n",
                        "Question: Explain the significance of the web app developed by the author and Robert. How did this innovation set them apart in the field of online store building?\n",
                        "Answer: The web app developed by the author and Robert allowed users to control the software by clicking on links through a browser, eliminating the need for any client software or command line inputs on the server. This innovation set them apart in the field of online store building by pioneering the concept of building and managing entire stores directly through a web browser, a method that was not widely recognized or utilized at the time. This approach streamlined the process for users and showcased the potential of web-based applications for creating and managing online stores without the need for additional software installations or technical expertise.\n",
                        "\n",
                        "Question: Why did the launch of Viaweb get delayed from the original plan in September?\n",
                        "Answer: The launch of Viaweb got delayed from the original plan in September because the founder realized halfway through the summer that he did not want to run a company, especially a big one. This realization led him to reconsider his motivations and goals, ultimately deciding to build a subset of the project that could be done as an open source project instead of pursuing the original vision as a company.\n",
                        "\n",
                        "Question: How did the author's background in art influence the development of the online store builder software?\n",
                        "Answer: The author's background in art did not directly influence the development of the online store builder software.\n",
                        "\n",
                        "Question: Describe the unique characteristics of Robert and Trevor as mentioned in the text, and how did their individual traits contribute to the success of the project?\n",
                        "Answer: Robert and Trevor are described as highly independent-minded individuals with distinct personalities. Robert is portrayed as persistent and dedicated, as evidenced by his commitment to the project over several years despite initial frustrations. On the other hand, Trevor is depicted as unconventional and innovative, with a unique approach to organizing his life through notecards.\n",
                        "\n",
                        "Their individual traits complemented each other in the project's success. Robert's perseverance ensured that the project continued to progress, while Trevor's creativity and effectiveness as a hacker brought valuable technical skills to the team. Together, their independent thinking and diverse strengths contributed to the development of different components of the software, ultimately leading to a successful launch of the online store platform.\n",
                        "\n",
                        "Question: How did the startup differentiate itself from competitors in terms of pricing, and what accidental strategy did they employ to attract users?\n",
                        "Answer: The startup differentiated itself from competitors by offering a low price for their services, charging $100 a month for a small store and $300 a month for a big one. An accidental strategy they employed to attract users was building stores for them, even though the software was designed for users to create their own stores.\n",
                        "\n",
                        "Question: According to the context, what did the founder realize was the ultimate test of a startup's success?\n",
                        "Answer: According to the context, the founder realized that the growth rate is the ultimate test of a startup's success.\n",
                        "\n",
                        "Question: What factors contributed to the delay in the author's company reaching breakeven, and how did this impact their control over the company's destiny?\n",
                        "Answer: The delay in the author's company reaching breakeven was influenced by the decision to hire more employees, driven by both investor pressure and the trend during the Internet Bubble era. This expansion in the workforce led to increased operational costs and delayed profitability. As a result, the company remained dependent on investors for funding, limiting their control over the company's destiny. The lack of profitability and the need for continuous investment meant that the company had to adhere to investor expectations and decisions, ultimately impacting their autonomy and strategic direction.\n",
                        "\n",
                        "Question: How did the author feel when Yahoo bought their company, and what did they do with the newfound wealth?\n",
                        "Answer: The author felt like they had transitioned from a modest lifestyle to a more affluent one when Yahoo acquired their company. With the newfound wealth, the author purchased a yellow 1998 VW GTI, considering its leather seats as the most luxurious possession they owned.\n",
                        "\n",
                        "Question: Why did the author's boss at Yahoo have a long conversation with them when they announced they were leaving?\n",
                        "Answer: The author's boss at Yahoo had a long conversation with them when they announced they were leaving because the boss thought the author might be leaving to start a new startup, potentially taking people with them, considering the author's valuable stock options and the booming Internet Bubble at that time.\n",
                        "\n",
                        "Question: What challenges did the author face when they tried to pursue their passion for painting after becoming rich?\n",
                        "Answer: The author faced challenges in pursuing their passion for painting after becoming rich due to a lack of energy and ambition, not knowing many people in California, living in a remote location in the Santa Cruz Mountains, and feeling disconnected from the artistic community. Additionally, returning to New York and resuming their old life while being rich presented a strange and somewhat disorienting experience.\n",
                        "\n",
                        "Question: What idea did the narrator have in the spring of 2000, based on their experience with Viaweb?\n",
                        "Answer: The narrator had the idea to start a new company called Aspra, which was going to be an \"application service provider\" or ASP.\n",
                        "\n",
                        "Question: Why did the author decide to start a new company focused on building web apps, and who did they initially try to recruit to work on the project with them?\n",
                        "Answer: The author decided to start a new company focused on building web apps because they believed that web apps were the future and saw the potential in creating a web app for making web apps. Initially, the author tried to recruit Robert to work on the project with them, but Robert, who was a postdoc at MIT at the time, declined the offer.\n",
                        "\n",
                        "Question: What led the author to reconsider running a company and decide to work on a subset of the project as an open source project instead?\n",
                        "Answer: The author reconsidered running a company and decided to work on a subset of the project as an open source project instead because he realized he no longer needed money and questioned why he was pursuing the vision as a company. This shift in perspective prompted him to focus on a subset that could be achieved as an open source project, leading to the decision to move away from building a big company.\n",
                        "\n",
                        "Question: What was the new dialect of Lisp that the author and Dan started working on, and where did they work on it?\n",
                        "Answer: The new dialect of Lisp that the author and Dan started working on was called Arc, and they worked on it in a house in Cambridge.\n",
                        "\n",
                        "Question: How did the author's experience with publishing essays online impact their future work and writing goals?\n",
                        "Answer: The author's experience with publishing essays online encouraged them to continue writing essays despite the initial perception that online essays were not prestigious. This experience led the author to realize the potential of online publishing, allowing for a broader audience reach compared to traditional print publishing. As a result, the author decided to focus on writing essays and recognized the value of working on projects that may not be considered prestigious at the time.\n",
                        "\n",
                        "Question: According to the author, what is a sign that there is something real to be discovered in unprestigious types of work?\n",
                        "Answer: Finding oneself drawn to some kind of work despite its current lack of prestige is a sign that there is something real to be discovered in unprestigious types of work.\n",
                        "\n",
                        "Question: Describe the circumstances in which the author met Jessica Livingston and how it led to them going out.\n",
                        "Answer: The author met Jessica Livingston at a big party at his house in October 2003. The party was organized by a mutual friend of theirs, Maria Daniels, who had the clever idea of having three separate hosts invite their friends to one party. This setup ensured that for every guest, two thirds of the other guests would be people they didn't know but would likely get along with. Jessica was one of the guests at this party whom the author didn't know initially but ended up liking. A couple of days after the party, the author asked Jessica out.\n",
                        "\n",
                        "Question: How did the author use talks as a tool for writing essays and what was the motivation behind giving a talk to the Harvard Computer Society?\n",
                        "Answer: The author used talks as a tool for writing essays by finding that the prospect of having to stand up in front of a group of people and share valuable information was a great spur to the imagination. The motivation behind giving a talk to the Harvard Computer Society was to inform them about how to start a startup, with the hope that they might be able to avoid the mistakes the author had made in the past.\n",
                        "\n",
                        "Question: How did the author's conversation with Jessica and the collaboration with Robert and Trevor lead to the decision to start their own investment firm?\n",
                        "Answer: The author's conversation with Jessica and the collaboration with Robert and Trevor led to the decision to start their own investment firm because they were frustrated with the delay in decision-making by venture capitalists. They decided to take matters into their own hands and implement the ideas they had been discussing. The author funded the new investment firm, Jessica left her job to work for it, and Robert and Trevor joined as partners, combining their skills and experiences to create something new and innovative.\n",
                        "\n",
                        "Question: What was the unique approach taken by the narrator and their partners in becoming angel investors, and how did it differ from traditional VC firms and individual angels?\n",
                        "Answer: The unique approach taken by the narrator and their partners in becoming angel investors was to combine the aspects of traditional VC firms and individual angels. They aimed to make smaller seed investments like individual angels but also provide comprehensive support to startups like traditional VC firms. This approach differed from traditional VC firms that focused on large investments and from individual angels who typically made investments on the side without offering extensive support to founders.\n",
                        "\n",
                        "Question: How did the batch model of funding startups come about at YC, and what was the reasoning behind it?\n",
                        "Answer: The batch model of funding startups at YC came about by accident due to the founders' ignorance about investing. They wanted to gain experience as investors and thought that funding a group of startups all at once would be a good way to do so. The reasoning behind this approach was to provide a structured program where startups could receive intensive help and support, while also creating a community of founders who could understand and assist each other with the challenges they were facing.\n",
                        "\n",
                        "Question: How did the Summer Founders Program evolve from a casual idea to a more serious endeavor, and what was the selection process for choosing which groups to fund?\n",
                        "Answer: The Summer Founders Program evolved from a casual idea to a more serious endeavor as the number of applications exceeded expectations, with many coming from individuals who had already graduated or were about to graduate. The selection process for choosing which groups to fund involved inviting about 20 out of the 225 groups to interview in person, from which 8 were ultimately selected for funding based on their impressive qualities and bold decision to participate in the program.\n",
                        "\n",
                        "Question: Explain the funding model used for startups in the Summer Founders Program and discuss the benefits of funding startups in batches according to the author.\n",
                        "Answer: The funding model for startups in the Summer Founders Program involved investing $6k per founder, totaling $12k for a typical two-founder case, in exchange for a 6% stake in the company. This model was considered fair as it was based on previous deals and was seen as advantageous for the founders. Additionally, the author mentions that providing free air conditioners to the founders during the hot summer further enhanced the support offered.\n",
                        "\n",
                        "Funding startups in batches was seen as a scalable approach by the author. It allowed for intensive support to be provided to multiple startups simultaneously, creating a more efficient process for both the investors and the startups. Moreover, being part of a batch was beneficial for the startups as it helped alleviate the common issue of isolation faced by founders. The batch model facilitated a sense of community and collaboration among the startups, providing them with a network of peers and resources to draw upon.\n",
                        "\n",
                        "Question: What unintended consequences arose as Y Combinator (YC) grew, particularly in terms of the community and customer base among startups?\n",
                        "Answer: As Y Combinator (YC) grew, unintended consequences arose in terms of the community and customer base among startups. The alumni formed a tight-knit community dedicated to helping one another, especially the current batch, creating a supportive network for founders. Additionally, startups within YC began to become each other's customers, leading to a situation where many startups obtained their initial set of customers primarily from their batchmates. This interconnectedness and support system among startups within YC grew stronger as the organization expanded.\n",
                        "\n",
                        "Question: What was the biggest source of stress for the author in relation to their work on YC and Hacker News?\n",
                        "Answer: The biggest source of stress for the author in relation to their work on YC and Hacker News was Hacker News (HN).\n",
                        "\n",
                        "Question: Who offered the author unsolicited advice during a visit to California in 2010?\n",
                        "Answer: Robert and Trevor\n",
                        "\n",
                        "Question: How did the author's perspective on Y Combinator change after receiving advice from Robert Morris, and what event ultimately led to the author considering leaving the organization?\n",
                        "Answer: The author's perspective on Y Combinator changed after receiving advice from Robert Morris as it made the author realize that Y Combinator might end up being the last significant project they worked on if they continued on their current trajectory. The event that ultimately led to the author considering leaving the organization was their mother having a stroke in the summer of 2012, caused by a blood clot from colon cancer.\n",
                        "\n",
                        "Question: How did the speaker's mother's health condition play a role in the decision to make changes in the leadership of Y Combinator?\n",
                        "Answer: The speaker's mother's health condition, specifically her stroke caused by colon cancer, prompted the speaker to reflect on his priorities and the time he was dedicating to Y Combinator. This reflection led to the decision to hand over the leadership of Y Combinator to someone else, ultimately resulting in the recruitment of Sam Altman as the new president.\n",
                        "\n",
                        "Question: What did the author decide to do after leaving YC in March, and how did they spend most of the rest of 2014?\n",
                        "Answer: After leaving YC in March, the author decided to focus on painting. They spent most of the rest of 2014 painting, aiming to see how good they could become by dedicating their time and effort to it.\n",
                        "\n",
                        "Question: Explain the significance of Lisp as a programming language and its core concept as described in the text.\n",
                        "Answer: Lisp is significant as a programming language due to its unique core concept of being defined by writing an interpreter in itself. Originally conceived as a formal model of computation rather than a traditional programming language, Lisp's core idea was to have the minimum set of predefined operators needed to write an interpreter for a language within the language itself. This approach, pioneered by John McCarthy, gave Lisp a power and elegance that set it apart from other languages. Despite its humble beginnings, Lisp evolved into a full-fledged programming language with added features over time, while still retaining its foundational concept of self-interpretation.\n",
                        "\n",
                        "Question: What challenges did McCarthy face in testing his interpreter for Lisp and how were they overcome?\n",
                        "Answer: McCarthy faced challenges in testing his interpreter for Lisp as it was becoming difficult to hand-simulate the execution of programs due to the increasing complexity. To overcome this limitation, computers were eventually used to run the interpreter, as they became powerful enough to handle the task. This shift allowed for more complicated interpreters to be tested effectively, enabling the development and refinement of the Lisp programming language.\n",
                        "\n",
                        "Question: Describe the author's approach to developing the new Lisp, Bel, and the difficulties encountered during the process.\n",
                        "Answer: The author developed the new Lisp, Bel, by writing it in Arc, which required a series of clever tricks to make it functional. The author had to refrain from writing essays to focus on the project, as the complexity of working on an interpreter written in itself made it challenging to keep track of errors and code levels. The process took 4 years, during which the author faced the difficulty of understanding the convoluted code after taking breaks from working on Bel. Despite the challenges, the author persisted and eventually completed Bel, following a precise goal to maintain motivation throughout the development process.\n",
                        "\n",
                        "Question: Why did the author ban himself from writing essays during the time he was working on Bel?\n",
                        "Answer: The author banned himself from writing essays during the time he was working on Bel because he felt that engaging in essay writing would have hindered his progress on the project.\n",
                        "\n",
                        "Question: What is Bel, and how does it compare to McCarthy's original Lisp in terms of its nature as a spec rather than an implementation?\n",
                        "Answer: Bel is a new Lisp language that was developed over a period of 4 years. Similar to McCarthy's original Lisp, Bel is a specification rather than an implementation. McCarthy's Lisp was also a spec expressed as code. The development of Bel involved using McCarthy's axiomatic approach to define a complete programming language, ensuring that every change made to McCarthy's Lisp was a discoveredness-preserving transformation. This approach aimed to result in a complete language of high quality.\n",
                        "\n",
                        "Question: Discuss the impact of Moore's Law on companies like Interleaf, as explained in note [5]. How did the exponential growth in processor power affect these companies?\n",
                        "Answer: The exponential growth in processor power, driven by Moore's Law, had a significant impact on companies like Interleaf. Despite having smart people and impressive technology, these companies were ultimately crushed by the rapid advancements in commodity processors, particularly Intel processors. The exponential growth in processor power during the 1990s essentially overshadowed high-end, special-purpose hardware and software companies like Interleaf, making it challenging for them to compete effectively in the market.\n",
                        "\n",
                        "Question: Discuss the relationship between money and coolness in the art world, as mentioned in the context information.\n",
                        "Answer: In the art world, there is a common perception that still life painting is considered the least prestigious form of painting. Similarly, the context suggests that certain types of work or pursuits may not always be perceived as cool or prestigious initially. The text implies that pursuing work or art forms that are not currently in the spotlight of prestige can lead to genuine discoveries and may indicate the right kind of motives. It suggests that being drawn to work that lacks current prestige can be a sign of authenticity and the absence of a desire to impress others. This highlights a relationship between money and coolness in the art world, where the pursuit of less prestigious forms of art or work may not be financially rewarding or socially esteemed initially, but can lead to meaningful discoveries and personal fulfillment.\n",
                        "\n",
                        "Question: Discuss the lesson learned from the author's experience with Y Combinator and how it relates to customs and rapid change in various fields. Provide examples to support your answer.\n",
                        "Answer: The lesson learned from the author's experience with Y Combinator is that customs can persist long after the original constraints that gave rise to them have disappeared. This is evident in the case of customary VC practices and the customs surrounding publishing essays, which were initially based on real constraints but have not evolved to reflect the changing landscape of startups and writing.\n",
                        "\n",
                        "In fields affected by rapid change, such as software and venture capital, customs rooted in outdated constraints can hinder progress and innovation. Being independent-minded and less influenced by these customs can provide an advantage in such rapidly changing fields. For example, the shift towards cheaper and more common startups has not been fully reflected in VC practices, highlighting the need for a more adaptable approach.\n",
                        "\n",
                        "The author's experience with Y Combinator also underscores the importance of challenging traditional customs and embracing change. By questioning established norms and being open to new ideas, individuals and organizations can better navigate evolving industries and capitalize on emerging opportunities. This flexibility and willingness to break away from outdated customs are essential for success in dynamic environments where predictability is limited.\n",
                        "\n",
                        "Question: Why does the author dislike the term 'deal flow' and what is the purpose of Y Combinator in relation to it?\n",
                        "Answer: The author dislikes the term 'deal flow' because it implies that the number of new startups at any given time is fixed, which the author believes is false. The purpose of Y Combinator is to falsify this notion by causing startups to be founded that would not otherwise have existed.\n",
                        "\n",
                        "Question: How does the author describe the impact of leaving YC, particularly in relation to their working relationship with Jessica?\n",
                        "Answer: The author describes the impact of leaving YC as a significant decision, especially in terms of their working relationship with Jessica.\n",
                        "\n",
                        "Question: Why is it mentioned that there may exist at least one path out of McCarthy's Lisp along which discoveredness is preserved?\n",
                        "Answer: Presumably aliens need numbers and errors and I/O too.\n",
                        "\n"
                    ]
                }
            ],
            "source": [
                "# loop over the questions and generate the answers\n",
                "for _, row in questions_with_document_chunk_df.iterrows():\n",
                "    question = row[\"question\"]\n",
                "    response_vector = query_engine.query(question)\n",
                "    print(f\"Question: {question}\\nAnswer: {response_vector.response}\\n\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now that we have executed the queries, we can start validating whether or not the RAG system was able to retrieve the correct context. Let's extract all the retrieved documents from the traces logged to phoenix. (For an in-depth explanation of how to export trace data from the phoenix runtime, consult the [docs](https://docs.arize.com/phoenix/how-to/extract-data-from-spans))."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 51,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th>context.trace_id</th>\n",
                            "      <th>input</th>\n",
                            "      <th>reference</th>\n",
                            "      <th>document_score</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th>document_position</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">3a9d75a6685a38c4</th>\n",
                            "      <th>0</th>\n",
                            "      <td>d43eb98002bb650e55c70374561ff8f1</td>\n",
                            "      <td>Why is it mentioned that there may exist at le...</td>\n",
                            "      <td>And at 50 there was some opportunity cost to s...</td>\n",
                            "      <td>0.856829</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>d43eb98002bb650e55c70374561ff8f1</td>\n",
                            "      <td>Why is it mentioned that there may exist at le...</td>\n",
                            "      <td>Individually these two phenomena are tedious b...</td>\n",
                            "      <td>0.851461</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">5dfcd3e2bd17873b</th>\n",
                            "      <th>0</th>\n",
                            "      <td>1757222d5ef48007db5b6ee222745495</td>\n",
                            "      <td>How does the author describe the impact of lea...</td>\n",
                            "      <td>Surely the biggest source of stress in one's w...</td>\n",
                            "      <td>0.841630</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>1757222d5ef48007db5b6ee222745495</td>\n",
                            "      <td>How does the author describe the impact of lea...</td>\n",
                            "      <td>\"You know,\" he said, \"you should make sure Y C...</td>\n",
                            "      <td>0.837039</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1d8524bf87f53687</th>\n",
                            "      <th>0</th>\n",
                            "      <td>ab80fe26411d35aedd87816a29017d1a</td>\n",
                            "      <td>Why does the author dislike the term 'deal flo...</td>\n",
                            "      <td>The YC logo itself is an inside joke: the Viaw...</td>\n",
                            "      <td>0.843317</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>...</th>\n",
                            "      <th>...</th>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>b71599449b06684b</th>\n",
                            "      <th>1</th>\n",
                            "      <td>375e1f36f66f3e0808d0cea84ab7466b</td>\n",
                            "      <td>What was the author's first experience with pr...</td>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "      <td>0.870292</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">8304b5cffaf1ee80</th>\n",
                            "      <th>0</th>\n",
                            "      <td>ad8dfdd419b2c262f06cbe18a0a1af22</td>\n",
                            "      <td>What was the author's experience with programm...</td>\n",
                            "      <td>I was puzzled by the 1401. I couldn't figure o...</td>\n",
                            "      <td>0.893548</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>ad8dfdd419b2c262f06cbe18a0a1af22</td>\n",
                            "      <td>What was the author's experience with programm...</td>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "      <td>0.880519</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">ded386711d92c952</th>\n",
                            "      <th>0</th>\n",
                            "      <td>9dc0d9a3148eb0b405bf73db7d7cc89c</td>\n",
                            "      <td>What were the two main things the author worke...</td>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "      <td>0.843013</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>9dc0d9a3148eb0b405bf73db7d7cc89c</td>\n",
                            "      <td>What were the two main things the author worke...</td>\n",
                            "      <td>Even then it took me several years to understa...</td>\n",
                            "      <td>0.816021</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>366 rows × 4 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                                    context.trace_id  \\\n",
                            "context.span_id  document_position                                     \n",
                            "3a9d75a6685a38c4 0                  d43eb98002bb650e55c70374561ff8f1   \n",
                            "                 1                  d43eb98002bb650e55c70374561ff8f1   \n",
                            "5dfcd3e2bd17873b 0                  1757222d5ef48007db5b6ee222745495   \n",
                            "                 1                  1757222d5ef48007db5b6ee222745495   \n",
                            "1d8524bf87f53687 0                  ab80fe26411d35aedd87816a29017d1a   \n",
                            "...                                                              ...   \n",
                            "b71599449b06684b 1                  375e1f36f66f3e0808d0cea84ab7466b   \n",
                            "8304b5cffaf1ee80 0                  ad8dfdd419b2c262f06cbe18a0a1af22   \n",
                            "                 1                  ad8dfdd419b2c262f06cbe18a0a1af22   \n",
                            "ded386711d92c952 0                  9dc0d9a3148eb0b405bf73db7d7cc89c   \n",
                            "                 1                  9dc0d9a3148eb0b405bf73db7d7cc89c   \n",
                            "\n",
                            "                                                                                input  \\\n",
                            "context.span_id  document_position                                                      \n",
                            "3a9d75a6685a38c4 0                  Why is it mentioned that there may exist at le...   \n",
                            "                 1                  Why is it mentioned that there may exist at le...   \n",
                            "5dfcd3e2bd17873b 0                  How does the author describe the impact of lea...   \n",
                            "                 1                  How does the author describe the impact of lea...   \n",
                            "1d8524bf87f53687 0                  Why does the author dislike the term 'deal flo...   \n",
                            "...                                                                               ...   \n",
                            "b71599449b06684b 1                  What was the author's first experience with pr...   \n",
                            "8304b5cffaf1ee80 0                  What was the author's experience with programm...   \n",
                            "                 1                  What was the author's experience with programm...   \n",
                            "ded386711d92c952 0                  What were the two main things the author worke...   \n",
                            "                 1                  What were the two main things the author worke...   \n",
                            "\n",
                            "                                                                            reference  \\\n",
                            "context.span_id  document_position                                                      \n",
                            "3a9d75a6685a38c4 0                  And at 50 there was some opportunity cost to s...   \n",
                            "                 1                  Individually these two phenomena are tedious b...   \n",
                            "5dfcd3e2bd17873b 0                  Surely the biggest source of stress in one's w...   \n",
                            "                 1                  \"You know,\" he said, \"you should make sure Y C...   \n",
                            "1d8524bf87f53687 0                  The YC logo itself is an inside joke: the Viaw...   \n",
                            "...                                                                               ...   \n",
                            "b71599449b06684b 1                  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...   \n",
                            "8304b5cffaf1ee80 0                  I was puzzled by the 1401. I couldn't figure o...   \n",
                            "                 1                  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...   \n",
                            "ded386711d92c952 0                  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...   \n",
                            "                 1                  Even then it took me several years to understa...   \n",
                            "\n",
                            "                                    document_score  \n",
                            "context.span_id  document_position                  \n",
                            "3a9d75a6685a38c4 0                        0.856829  \n",
                            "                 1                        0.851461  \n",
                            "5dfcd3e2bd17873b 0                        0.841630  \n",
                            "                 1                        0.837039  \n",
                            "1d8524bf87f53687 0                        0.843317  \n",
                            "...                                            ...  \n",
                            "b71599449b06684b 1                        0.870292  \n",
                            "8304b5cffaf1ee80 0                        0.893548  \n",
                            "                 1                        0.880519  \n",
                            "ded386711d92c952 0                        0.843013  \n",
                            "                 1                        0.816021  \n",
                            "\n",
                            "[366 rows x 4 columns]"
                        ]
                    },
                    "execution_count": 51,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "from phoenix.session.evaluation import get_retrieved_documents\n",
                "\n",
                "retrieved_documents_df = get_retrieved_documents(px.Client())\n",
                "retrieved_documents_df"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's now use Phoenix's LLM Evals to evaluate the relevance of the retrieved documents with regards to the query. Note, we've turned on `explanations` which prompts the LLM to explain it's reasoning. This can be useful for debugging and for figuring out potential corrective actions."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 52,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "application/vnd.jupyter.widget-view+json": {
                            "model_id": "cbe61513c0e843748c56ccf2a2acea8f",
                            "version_major": 2,
                            "version_minor": 0
                        },
                        "text/plain": [
                            "run_evals |          | 0/366 (0.0%) | ⏳ 00:00<? | ?it/s"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                }
            ],
            "source": [
                "from phoenix.evals import (\n",
                "    RelevanceEvaluator,\n",
                "    run_evals,\n",
                ")\n",
                "\n",
                "relevance_evaluator = RelevanceEvaluator(OpenAIModel(model=\"gpt-4-turbo-preview\"))\n",
                "\n",
                "retrieved_documents_relevance_df = run_evals(\n",
                "    evaluators=[relevance_evaluator],\n",
                "    dataframe=retrieved_documents_df,\n",
                "    provide_explanation=True,\n",
                "    concurrency=20,\n",
                ")[0]"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 53,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th>label</th>\n",
                            "      <th>score</th>\n",
                            "      <th>explanation</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th>document_position</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">3a9d75a6685a38c4</th>\n",
                            "      <th>0</th>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks why it is mentioned that the...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks why it is mentioned that the...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">5dfcd3e2bd17873b</th>\n",
                            "      <th>0</th>\n",
                            "      <td>unrelated</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The question asks about the author's descripti...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks about the author's descripti...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1d8524bf87f53687</th>\n",
                            "      <th>0</th>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The reference text directly addresses the ques...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                        label  score  \\\n",
                            "context.span_id  document_position                     \n",
                            "3a9d75a6685a38c4 0                   relevant      1   \n",
                            "                 1                   relevant      1   \n",
                            "5dfcd3e2bd17873b 0                  unrelated      0   \n",
                            "                 1                   relevant      1   \n",
                            "1d8524bf87f53687 0                   relevant      1   \n",
                            "\n",
                            "                                                                          explanation  \n",
                            "context.span_id  document_position                                                     \n",
                            "3a9d75a6685a38c4 0                  The question asks why it is mentioned that the...  \n",
                            "                 1                  The question asks why it is mentioned that the...  \n",
                            "5dfcd3e2bd17873b 0                  The question asks about the author's descripti...  \n",
                            "                 1                  The question asks about the author's descripti...  \n",
                            "1d8524bf87f53687 0                  The reference text directly addresses the ques...  "
                        ]
                    },
                    "execution_count": 53,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "retrieved_documents_relevance_df.head()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We can now combine the documents with the relevance evaluations to compute retrieval metrics. These metrics will help us understand how well the RAG system is performing."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 54,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th>context.trace_id</th>\n",
                            "      <th>input</th>\n",
                            "      <th>reference</th>\n",
                            "      <th>document_score</th>\n",
                            "      <th>eval_label</th>\n",
                            "      <th>eval_score</th>\n",
                            "      <th>eval_explanation</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th>document_position</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">3a9d75a6685a38c4</th>\n",
                            "      <th>0</th>\n",
                            "      <td>d43eb98002bb650e55c70374561ff8f1</td>\n",
                            "      <td>Why is it mentioned that there may exist at le...</td>\n",
                            "      <td>And at 50 there was some opportunity cost to s...</td>\n",
                            "      <td>0.856829</td>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks why it is mentioned that the...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>d43eb98002bb650e55c70374561ff8f1</td>\n",
                            "      <td>Why is it mentioned that there may exist at le...</td>\n",
                            "      <td>Individually these two phenomena are tedious b...</td>\n",
                            "      <td>0.851461</td>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks why it is mentioned that the...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">5dfcd3e2bd17873b</th>\n",
                            "      <th>0</th>\n",
                            "      <td>1757222d5ef48007db5b6ee222745495</td>\n",
                            "      <td>How does the author describe the impact of lea...</td>\n",
                            "      <td>Surely the biggest source of stress in one's w...</td>\n",
                            "      <td>0.841630</td>\n",
                            "      <td>unrelated</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The question asks about the author's descripti...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>1757222d5ef48007db5b6ee222745495</td>\n",
                            "      <td>How does the author describe the impact of lea...</td>\n",
                            "      <td>\"You know,\" he said, \"you should make sure Y C...</td>\n",
                            "      <td>0.837039</td>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks about the author's descripti...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1d8524bf87f53687</th>\n",
                            "      <th>0</th>\n",
                            "      <td>ab80fe26411d35aedd87816a29017d1a</td>\n",
                            "      <td>Why does the author dislike the term 'deal flo...</td>\n",
                            "      <td>The YC logo itself is an inside joke: the Viaw...</td>\n",
                            "      <td>0.843317</td>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The reference text directly addresses the ques...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>...</th>\n",
                            "      <th>...</th>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>b71599449b06684b</th>\n",
                            "      <th>1</th>\n",
                            "      <td>375e1f36f66f3e0808d0cea84ab7466b</td>\n",
                            "      <td>What was the author's first experience with pr...</td>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "      <td>0.870292</td>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks for two specific pieces of i...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">8304b5cffaf1ee80</th>\n",
                            "      <th>0</th>\n",
                            "      <td>ad8dfdd419b2c262f06cbe18a0a1af22</td>\n",
                            "      <td>What was the author's experience with programm...</td>\n",
                            "      <td>I was puzzled by the 1401. I couldn't figure o...</td>\n",
                            "      <td>0.893548</td>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The reference text directly addresses the ques...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>ad8dfdd419b2c262f06cbe18a0a1af22</td>\n",
                            "      <td>What was the author's experience with programm...</td>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "      <td>0.880519</td>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The reference text directly addresses the ques...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th rowspan=\"2\" valign=\"top\">ded386711d92c952</th>\n",
                            "      <th>0</th>\n",
                            "      <td>9dc0d9a3148eb0b405bf73db7d7cc89c</td>\n",
                            "      <td>What were the two main things the author worke...</td>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "      <td>0.843013</td>\n",
                            "      <td>relevant</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks about the two main activitie...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>9dc0d9a3148eb0b405bf73db7d7cc89c</td>\n",
                            "      <td>What were the two main things the author worke...</td>\n",
                            "      <td>Even then it took me several years to understa...</td>\n",
                            "      <td>0.816021</td>\n",
                            "      <td>unrelated</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The question asks about the two main things th...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>366 rows × 7 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                                    context.trace_id  \\\n",
                            "context.span_id  document_position                                     \n",
                            "3a9d75a6685a38c4 0                  d43eb98002bb650e55c70374561ff8f1   \n",
                            "                 1                  d43eb98002bb650e55c70374561ff8f1   \n",
                            "5dfcd3e2bd17873b 0                  1757222d5ef48007db5b6ee222745495   \n",
                            "                 1                  1757222d5ef48007db5b6ee222745495   \n",
                            "1d8524bf87f53687 0                  ab80fe26411d35aedd87816a29017d1a   \n",
                            "...                                                              ...   \n",
                            "b71599449b06684b 1                  375e1f36f66f3e0808d0cea84ab7466b   \n",
                            "8304b5cffaf1ee80 0                  ad8dfdd419b2c262f06cbe18a0a1af22   \n",
                            "                 1                  ad8dfdd419b2c262f06cbe18a0a1af22   \n",
                            "ded386711d92c952 0                  9dc0d9a3148eb0b405bf73db7d7cc89c   \n",
                            "                 1                  9dc0d9a3148eb0b405bf73db7d7cc89c   \n",
                            "\n",
                            "                                                                                input  \\\n",
                            "context.span_id  document_position                                                      \n",
                            "3a9d75a6685a38c4 0                  Why is it mentioned that there may exist at le...   \n",
                            "                 1                  Why is it mentioned that there may exist at le...   \n",
                            "5dfcd3e2bd17873b 0                  How does the author describe the impact of lea...   \n",
                            "                 1                  How does the author describe the impact of lea...   \n",
                            "1d8524bf87f53687 0                  Why does the author dislike the term 'deal flo...   \n",
                            "...                                                                               ...   \n",
                            "b71599449b06684b 1                  What was the author's first experience with pr...   \n",
                            "8304b5cffaf1ee80 0                  What was the author's experience with programm...   \n",
                            "                 1                  What was the author's experience with programm...   \n",
                            "ded386711d92c952 0                  What were the two main things the author worke...   \n",
                            "                 1                  What were the two main things the author worke...   \n",
                            "\n",
                            "                                                                            reference  \\\n",
                            "context.span_id  document_position                                                      \n",
                            "3a9d75a6685a38c4 0                  And at 50 there was some opportunity cost to s...   \n",
                            "                 1                  Individually these two phenomena are tedious b...   \n",
                            "5dfcd3e2bd17873b 0                  Surely the biggest source of stress in one's w...   \n",
                            "                 1                  \"You know,\" he said, \"you should make sure Y C...   \n",
                            "1d8524bf87f53687 0                  The YC logo itself is an inside joke: the Viaw...   \n",
                            "...                                                                               ...   \n",
                            "b71599449b06684b 1                  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...   \n",
                            "8304b5cffaf1ee80 0                  I was puzzled by the 1401. I couldn't figure o...   \n",
                            "                 1                  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...   \n",
                            "ded386711d92c952 0                  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...   \n",
                            "                 1                  Even then it took me several years to understa...   \n",
                            "\n",
                            "                                    document_score eval_label  eval_score  \\\n",
                            "context.span_id  document_position                                          \n",
                            "3a9d75a6685a38c4 0                        0.856829   relevant           1   \n",
                            "                 1                        0.851461   relevant           1   \n",
                            "5dfcd3e2bd17873b 0                        0.841630  unrelated           0   \n",
                            "                 1                        0.837039   relevant           1   \n",
                            "1d8524bf87f53687 0                        0.843317   relevant           1   \n",
                            "...                                            ...        ...         ...   \n",
                            "b71599449b06684b 1                        0.870292   relevant           1   \n",
                            "8304b5cffaf1ee80 0                        0.893548   relevant           1   \n",
                            "                 1                        0.880519   relevant           1   \n",
                            "ded386711d92c952 0                        0.843013   relevant           1   \n",
                            "                 1                        0.816021  unrelated           0   \n",
                            "\n",
                            "                                                                     eval_explanation  \n",
                            "context.span_id  document_position                                                     \n",
                            "3a9d75a6685a38c4 0                  The question asks why it is mentioned that the...  \n",
                            "                 1                  The question asks why it is mentioned that the...  \n",
                            "5dfcd3e2bd17873b 0                  The question asks about the author's descripti...  \n",
                            "                 1                  The question asks about the author's descripti...  \n",
                            "1d8524bf87f53687 0                  The reference text directly addresses the ques...  \n",
                            "...                                                                               ...  \n",
                            "b71599449b06684b 1                  The question asks for two specific pieces of i...  \n",
                            "8304b5cffaf1ee80 0                  The reference text directly addresses the ques...  \n",
                            "                 1                  The reference text directly addresses the ques...  \n",
                            "ded386711d92c952 0                  The question asks about the two main activitie...  \n",
                            "                 1                  The question asks about the two main things th...  \n",
                            "\n",
                            "[366 rows x 7 columns]"
                        ]
                    },
                    "execution_count": 54,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "documents_with_relevance_df = pd.concat(\n",
                "    [retrieved_documents_df, retrieved_documents_relevance_df.add_prefix(\"eval_\")], axis=1\n",
                ")\n",
                "documents_with_relevance_df"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's compute Normalized Discounted Cumulative Gain [NCDG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) at 2 for all our retrieval steps.  In information retrieval, this metric is often used to measure effectiveness of search engine algorithms and related applications."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 55,
            "metadata": {},
            "outputs": [],
            "source": [
                "import numpy as np\n",
                "from sklearn.metrics import ndcg_score\n",
                "\n",
                "\n",
                "def _compute_ndcg(df: pd.DataFrame, k: int):\n",
                "    \"\"\"Compute NDCG@k in the presence of missing values\"\"\"\n",
                "    n = max(2, len(df))\n",
                "    eval_scores = np.zeros(n)\n",
                "    doc_scores = np.zeros(n)\n",
                "    eval_scores[: len(df)] = df.eval_score\n",
                "    doc_scores[: len(df)] = df.document_score\n",
                "    try:\n",
                "        return ndcg_score([eval_scores], [doc_scores], k=k)\n",
                "    except ValueError:\n",
                "        return np.nan\n",
                "\n",
                "\n",
                "ndcg_at_2 = pd.DataFrame(\n",
                "    {\"score\": documents_with_relevance_df.groupby(\"context.span_id\").apply(_compute_ndcg, k=2)}\n",
                ")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 56,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>score</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>026b2abd5dcb9849</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>026c709c4b5a4e09</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>03e2f09afc0aa0b1</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>05234f3093317c5f</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>05eee051d13f2a7d</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>...</th>\n",
                            "      <td>...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fca2541676815e1a</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fd44acf228788f3b</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fdecfc1e376821de</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fee739fffff0cf58</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fefe38ad43b7ea5f</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>183 rows × 1 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                  score\n",
                            "context.span_id        \n",
                            "026b2abd5dcb9849    1.0\n",
                            "026c709c4b5a4e09    1.0\n",
                            "03e2f09afc0aa0b1    1.0\n",
                            "05234f3093317c5f    1.0\n",
                            "05eee051d13f2a7d    1.0\n",
                            "...                 ...\n",
                            "fca2541676815e1a    1.0\n",
                            "fd44acf228788f3b    1.0\n",
                            "fdecfc1e376821de    1.0\n",
                            "fee739fffff0cf58    1.0\n",
                            "fefe38ad43b7ea5f    1.0\n",
                            "\n",
                            "[183 rows x 1 columns]"
                        ]
                    },
                    "execution_count": 56,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "ndcg_at_2"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's also compute precision at 2 for all our retrieval steps."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 57,
            "metadata": {},
            "outputs": [],
            "source": [
                "precision_at_2 = pd.DataFrame(\n",
                "    {\n",
                "        \"score\": documents_with_relevance_df.groupby(\"context.span_id\").apply(\n",
                "            lambda x: x.eval_score[:2].sum(skipna=False) / 2\n",
                "        )\n",
                "    }\n",
                ")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 58,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>score</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>026b2abd5dcb9849</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>026c709c4b5a4e09</th>\n",
                            "      <td>0.5</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>03e2f09afc0aa0b1</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>05234f3093317c5f</th>\n",
                            "      <td>0.5</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>05eee051d13f2a7d</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>...</th>\n",
                            "      <td>...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fca2541676815e1a</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fd44acf228788f3b</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fdecfc1e376821de</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fee739fffff0cf58</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fefe38ad43b7ea5f</th>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>183 rows × 1 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                  score\n",
                            "context.span_id        \n",
                            "026b2abd5dcb9849    1.0\n",
                            "026c709c4b5a4e09    0.5\n",
                            "03e2f09afc0aa0b1    1.0\n",
                            "05234f3093317c5f    0.5\n",
                            "05eee051d13f2a7d    1.0\n",
                            "...                 ...\n",
                            "fca2541676815e1a    1.0\n",
                            "fd44acf228788f3b    1.0\n",
                            "fdecfc1e376821de    1.0\n",
                            "fee739fffff0cf58    1.0\n",
                            "fefe38ad43b7ea5f    1.0\n",
                            "\n",
                            "[183 rows x 1 columns]"
                        ]
                    },
                    "execution_count": 58,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "precision_at_2"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Lastly, let's compute whether or not a correct document was retrieved at all for each query (e.g. a hit)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 59,
            "metadata": {},
            "outputs": [],
            "source": [
                "hit = pd.DataFrame(\n",
                "    {\n",
                "        \"hit\": documents_with_relevance_df.groupby(\"context.span_id\").apply(\n",
                "            lambda x: x.eval_score[:2].sum(skipna=False) > 0\n",
                "        )\n",
                "    }\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's now view the results in a combined dataframe."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 60,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>attributes.input.value</th>\n",
                            "      <th>ncdg@2_score</th>\n",
                            "      <th>precision@2_score</th>\n",
                            "      <th>hit</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>3a9d75a6685a38c4</th>\n",
                            "      <td>Why is it mentioned that there may exist at le...</td>\n",
                            "      <td>1.00000</td>\n",
                            "      <td>1.0</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>5dfcd3e2bd17873b</th>\n",
                            "      <td>How does the author describe the impact of lea...</td>\n",
                            "      <td>0.63093</td>\n",
                            "      <td>0.5</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1d8524bf87f53687</th>\n",
                            "      <td>Why does the author dislike the term 'deal flo...</td>\n",
                            "      <td>1.00000</td>\n",
                            "      <td>1.0</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>43c475deb5849b80</th>\n",
                            "      <td>Discuss the lesson learned from the author's e...</td>\n",
                            "      <td>1.00000</td>\n",
                            "      <td>1.0</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4640d0a90cc272e5</th>\n",
                            "      <td>Discuss the relationship between money and coo...</td>\n",
                            "      <td>0.00000</td>\n",
                            "      <td>0.0</td>\n",
                            "      <td>False</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>...</th>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3b86bcd4bcd0a600</th>\n",
                            "      <td>What was the default language at Cornell and o...</td>\n",
                            "      <td>1.00000</td>\n",
                            "      <td>0.5</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>f3205a8197cb8e6d</th>\n",
                            "      <td>What novel by Heinlein inspired the individual...</td>\n",
                            "      <td>1.00000</td>\n",
                            "      <td>1.0</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>b71599449b06684b</th>\n",
                            "      <td>What was the author's first experience with pr...</td>\n",
                            "      <td>1.00000</td>\n",
                            "      <td>1.0</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>8304b5cffaf1ee80</th>\n",
                            "      <td>What was the author's experience with programm...</td>\n",
                            "      <td>1.00000</td>\n",
                            "      <td>1.0</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>ded386711d92c952</th>\n",
                            "      <td>What were the two main things the author worke...</td>\n",
                            "      <td>1.00000</td>\n",
                            "      <td>0.5</td>\n",
                            "      <td>True</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>183 rows × 4 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                             attributes.input.value  \\\n",
                            "context.span_id                                                       \n",
                            "3a9d75a6685a38c4  Why is it mentioned that there may exist at le...   \n",
                            "5dfcd3e2bd17873b  How does the author describe the impact of lea...   \n",
                            "1d8524bf87f53687  Why does the author dislike the term 'deal flo...   \n",
                            "43c475deb5849b80  Discuss the lesson learned from the author's e...   \n",
                            "4640d0a90cc272e5  Discuss the relationship between money and coo...   \n",
                            "...                                                             ...   \n",
                            "3b86bcd4bcd0a600  What was the default language at Cornell and o...   \n",
                            "f3205a8197cb8e6d  What novel by Heinlein inspired the individual...   \n",
                            "b71599449b06684b  What was the author's first experience with pr...   \n",
                            "8304b5cffaf1ee80  What was the author's experience with programm...   \n",
                            "ded386711d92c952  What were the two main things the author worke...   \n",
                            "\n",
                            "                  ncdg@2_score  precision@2_score    hit  \n",
                            "context.span_id                                           \n",
                            "3a9d75a6685a38c4       1.00000                1.0   True  \n",
                            "5dfcd3e2bd17873b       0.63093                0.5   True  \n",
                            "1d8524bf87f53687       1.00000                1.0   True  \n",
                            "43c475deb5849b80       1.00000                1.0   True  \n",
                            "4640d0a90cc272e5       0.00000                0.0  False  \n",
                            "...                        ...                ...    ...  \n",
                            "3b86bcd4bcd0a600       1.00000                0.5   True  \n",
                            "f3205a8197cb8e6d       1.00000                1.0   True  \n",
                            "b71599449b06684b       1.00000                1.0   True  \n",
                            "8304b5cffaf1ee80       1.00000                1.0   True  \n",
                            "ded386711d92c952       1.00000                0.5   True  \n",
                            "\n",
                            "[183 rows x 4 columns]"
                        ]
                    },
                    "execution_count": 60,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "retrievals_df = px.Client().get_spans_dataframe(\"span_kind == 'RETRIEVER'\")\n",
                "rag_evaluation_dataframe = pd.concat(\n",
                "    [\n",
                "        retrievals_df[\"attributes.input.value\"],\n",
                "        ndcg_at_2.add_prefix(\"ncdg@2_\"),\n",
                "        precision_at_2.add_prefix(\"precision@2_\"),\n",
                "        hit,\n",
                "    ],\n",
                "    axis=1,\n",
                ")\n",
                "rag_evaluation_dataframe"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Observations\n",
                "\n",
                "Let's now take our results and aggregate them to get a sense of how well our RAG system is performing."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 61,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "ncdg@2_score         0.934685\n",
                            "precision@2_score    0.849727\n",
                            "hit                  0.950820\n",
                            "dtype: float64"
                        ]
                    },
                    "execution_count": 61,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Aggregate the scores across the retrievals\n",
                "results = rag_evaluation_dataframe.mean(numeric_only=True)\n",
                "results"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "As we can see from the above numbers, our RAG system is not perfect, there are times when it fails to retrieve the correct context within the first two documents. At other times the correct context is included in the top 2 results but non-relevant information is also included in the context. This is an indication that we need to improve our retrieval strategy. One possible solution could be to increase the number of documents retrieved and then use a more sophisticated ranking strategy (such as a reranker) to select the correct context."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We have now evaluated our RAG system's retrieval performance. Let's send these evaluations to Phoenix for visualization. By sending the evaluations to Phoenix, you will be able to view the evaluations alongside the traces that were captured earlier."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 62,
            "metadata": {},
            "outputs": [],
            "source": [
                "from phoenix.trace import DocumentEvaluations, SpanEvaluations\n",
                "\n",
                "px.Client().log_evaluations(\n",
                "    SpanEvaluations(dataframe=ndcg_at_2, eval_name=\"ndcg@2\"),\n",
                "    SpanEvaluations(dataframe=precision_at_2, eval_name=\"precision@2\"),\n",
                "    DocumentEvaluations(dataframe=retrieved_documents_relevance_df, eval_name=\"relevance\"),\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Response Evaluation\n",
                "\n",
                "The retrieval evaluations demonstrates that our RAG system is not perfect. However, it's possible that the LLM is able to generate the correct response even when the context is incorrect. Let's evaluate the responses generated by the LLM."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 63,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>input</th>\n",
                            "      <th>output</th>\n",
                            "      <th>reference</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0e920887bad0f2bc</th>\n",
                            "      <td>Why is it mentioned that there may exist at le...</td>\n",
                            "      <td>Presumably aliens need numbers and errors and ...</td>\n",
                            "      <td>And at 50 there was some opportunity cost to s...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>a424561f80b13e37</th>\n",
                            "      <td>How does the author describe the impact of lea...</td>\n",
                            "      <td>The author describes the impact of leaving YC ...</td>\n",
                            "      <td>Surely the biggest source of stress in one's w...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>b12bafbd789ea6df</th>\n",
                            "      <td>Why does the author dislike the term 'deal flo...</td>\n",
                            "      <td>The author dislikes the term 'deal flow' becau...</td>\n",
                            "      <td>The YC logo itself is an inside joke: the Viaw...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>ef38d55b343fb6c2</th>\n",
                            "      <td>Discuss the lesson learned from the author's e...</td>\n",
                            "      <td>The lesson learned from the author's experienc...</td>\n",
                            "      <td>Customary VC practice had once, like the custo...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>aaf2cd7f34c25184</th>\n",
                            "      <td>Discuss the relationship between money and coo...</td>\n",
                            "      <td>In the art world, there is a common perception...</td>\n",
                            "      <td>You want to emphasize the visual cues that tel...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>...</th>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "      <td>...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>471dd22d80405a26</th>\n",
                            "      <td>What was the default language at Cornell and o...</td>\n",
                            "      <td>The default language at Cornell and other univ...</td>\n",
                            "      <td>The default language at Cornell was a Pascal-l...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>20ead71f6c7ba0bb</th>\n",
                            "      <td>What novel by Heinlein inspired the individual...</td>\n",
                            "      <td>The Moon is a Harsh Mistress</td>\n",
                            "      <td>I couldn't have put this into words when I was...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>a9c939cbbb74461b</th>\n",
                            "      <td>What was the author's first experience with pr...</td>\n",
                            "      <td>The author's first experience with programming...</td>\n",
                            "      <td>I remember vividly how impressed and envious I...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>5f0ab100e5472478</th>\n",
                            "      <td>What was the author's experience with programm...</td>\n",
                            "      <td>The author's experience with programming on th...</td>\n",
                            "      <td>I was puzzled by the 1401. I couldn't figure o...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>758710d85d877d46</th>\n",
                            "      <td>What were the two main things the author worke...</td>\n",
                            "      <td>The author worked on writing and programming b...</td>\n",
                            "      <td>What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>183 rows × 3 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                                              input  \\\n",
                            "context.span_id                                                       \n",
                            "0e920887bad0f2bc  Why is it mentioned that there may exist at le...   \n",
                            "a424561f80b13e37  How does the author describe the impact of lea...   \n",
                            "b12bafbd789ea6df  Why does the author dislike the term 'deal flo...   \n",
                            "ef38d55b343fb6c2  Discuss the lesson learned from the author's e...   \n",
                            "aaf2cd7f34c25184  Discuss the relationship between money and coo...   \n",
                            "...                                                             ...   \n",
                            "471dd22d80405a26  What was the default language at Cornell and o...   \n",
                            "20ead71f6c7ba0bb  What novel by Heinlein inspired the individual...   \n",
                            "a9c939cbbb74461b  What was the author's first experience with pr...   \n",
                            "5f0ab100e5472478  What was the author's experience with programm...   \n",
                            "758710d85d877d46  What were the two main things the author worke...   \n",
                            "\n",
                            "                                                             output  \\\n",
                            "context.span_id                                                       \n",
                            "0e920887bad0f2bc  Presumably aliens need numbers and errors and ...   \n",
                            "a424561f80b13e37  The author describes the impact of leaving YC ...   \n",
                            "b12bafbd789ea6df  The author dislikes the term 'deal flow' becau...   \n",
                            "ef38d55b343fb6c2  The lesson learned from the author's experienc...   \n",
                            "aaf2cd7f34c25184  In the art world, there is a common perception...   \n",
                            "...                                                             ...   \n",
                            "471dd22d80405a26  The default language at Cornell and other univ...   \n",
                            "20ead71f6c7ba0bb                       The Moon is a Harsh Mistress   \n",
                            "a9c939cbbb74461b  The author's first experience with programming...   \n",
                            "5f0ab100e5472478  The author's experience with programming on th...   \n",
                            "758710d85d877d46  The author worked on writing and programming b...   \n",
                            "\n",
                            "                                                          reference  \n",
                            "context.span_id                                                      \n",
                            "0e920887bad0f2bc  And at 50 there was some opportunity cost to s...  \n",
                            "a424561f80b13e37  Surely the biggest source of stress in one's w...  \n",
                            "b12bafbd789ea6df  The YC logo itself is an inside joke: the Viaw...  \n",
                            "ef38d55b343fb6c2  Customary VC practice had once, like the custo...  \n",
                            "aaf2cd7f34c25184  You want to emphasize the visual cues that tel...  \n",
                            "...                                                             ...  \n",
                            "471dd22d80405a26  The default language at Cornell was a Pascal-l...  \n",
                            "20ead71f6c7ba0bb  I couldn't have put this into words when I was...  \n",
                            "a9c939cbbb74461b  I remember vividly how impressed and envious I...  \n",
                            "5f0ab100e5472478  I was puzzled by the 1401. I couldn't figure o...  \n",
                            "758710d85d877d46  What I Worked On\\n\\nFebruary 2021\\n\\nBefore co...  \n",
                            "\n",
                            "[183 rows x 3 columns]"
                        ]
                    },
                    "execution_count": 63,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "from phoenix.session.evaluation import get_qa_with_reference\n",
                "\n",
                "qa_with_reference_df = get_qa_with_reference(px.Client())\n",
                "qa_with_reference_df"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now that we have a dataset of the question, context, and response (input, reference, and output), we now can measure how well the LLM is responding to the queries. For details on the QA correctness evaluation, see the [LLM Evals documentation](https://docs.arize.com/phoenix/llm-evals/running-pre-tested-evals/q-and-a-on-retrieved-data)."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 64,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "application/vnd.jupyter.widget-view+json": {
                            "model_id": "56864cc5565c44a794347700481cde54",
                            "version_major": 2,
                            "version_minor": 0
                        },
                        "text/plain": [
                            "run_evals |          | 0/366 (0.0%) | ⏳ 00:00<? | ?it/s"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Exception in worker on attempt 1: raised InternalServerError('<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body>\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>cloudflare</center>\\r\\n</body>\\r\\n</html>')\n",
                        "Requeuing...\n"
                    ]
                }
            ],
            "source": [
                "from phoenix.evals import (\n",
                "    HallucinationEvaluator,\n",
                "    OpenAIModel,\n",
                "    QAEvaluator,\n",
                "    run_evals,\n",
                ")\n",
                "\n",
                "qa_evaluator = QAEvaluator(OpenAIModel(model=\"gpt-4-turbo-preview\"))\n",
                "hallucination_evaluator = HallucinationEvaluator(OpenAIModel(model=\"gpt-4-turbo-preview\"))\n",
                "\n",
                "qa_correctness_eval_df, hallucination_eval_df = run_evals(\n",
                "    evaluators=[qa_evaluator, hallucination_evaluator],\n",
                "    dataframe=qa_with_reference_df,\n",
                "    provide_explanation=True,\n",
                "    concurrency=20,\n",
                ")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 65,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>label</th>\n",
                            "      <th>score</th>\n",
                            "      <th>explanation</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0e920887bad0f2bc</th>\n",
                            "      <td>incorrect</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The question asks why it is mentioned that the...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>a424561f80b13e37</th>\n",
                            "      <td>incorrect</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The reference text provides a detailed account...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>b12bafbd789ea6df</th>\n",
                            "      <td>correct</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The question asks for two pieces of informatio...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>ef38d55b343fb6c2</th>\n",
                            "      <td>correct</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The given answer accurately captures the essen...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>aaf2cd7f34c25184</th>\n",
                            "      <td>incorrect</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The question asks about the relationship betwe...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                      label  score  \\\n",
                            "context.span_id                      \n",
                            "0e920887bad0f2bc  incorrect      0   \n",
                            "a424561f80b13e37  incorrect      0   \n",
                            "b12bafbd789ea6df    correct      1   \n",
                            "ef38d55b343fb6c2    correct      1   \n",
                            "aaf2cd7f34c25184  incorrect      0   \n",
                            "\n",
                            "                                                        explanation  \n",
                            "context.span_id                                                      \n",
                            "0e920887bad0f2bc  The question asks why it is mentioned that the...  \n",
                            "a424561f80b13e37  The reference text provides a detailed account...  \n",
                            "b12bafbd789ea6df  The question asks for two pieces of informatio...  \n",
                            "ef38d55b343fb6c2  The given answer accurately captures the essen...  \n",
                            "aaf2cd7f34c25184  The question asks about the relationship betwe...  "
                        ]
                    },
                    "execution_count": 65,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "qa_correctness_eval_df.head()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 66,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>label</th>\n",
                            "      <th>score</th>\n",
                            "      <th>explanation</th>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>context.span_id</th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "      <th></th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0e920887bad0f2bc</th>\n",
                            "      <td>factual</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The query asks why it is mentioned that there ...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>a424561f80b13e37</th>\n",
                            "      <td>hallucinated</td>\n",
                            "      <td>1</td>\n",
                            "      <td>The reference text provides detailed informati...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>b12bafbd789ea6df</th>\n",
                            "      <td>factual</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The answer provided directly reflects the info...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>ef38d55b343fb6c2</th>\n",
                            "      <td>factual</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The answer accurately reflects the content and...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>aaf2cd7f34c25184</th>\n",
                            "      <td>factual</td>\n",
                            "      <td>0</td>\n",
                            "      <td>The answer discusses the perception of still l...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                         label  score  \\\n",
                            "context.span_id                         \n",
                            "0e920887bad0f2bc       factual      0   \n",
                            "a424561f80b13e37  hallucinated      1   \n",
                            "b12bafbd789ea6df       factual      0   \n",
                            "ef38d55b343fb6c2       factual      0   \n",
                            "aaf2cd7f34c25184       factual      0   \n",
                            "\n",
                            "                                                        explanation  \n",
                            "context.span_id                                                      \n",
                            "0e920887bad0f2bc  The query asks why it is mentioned that there ...  \n",
                            "a424561f80b13e37  The reference text provides detailed informati...  \n",
                            "b12bafbd789ea6df  The answer provided directly reflects the info...  \n",
                            "ef38d55b343fb6c2  The answer accurately reflects the content and...  \n",
                            "aaf2cd7f34c25184  The answer discusses the perception of still l...  "
                        ]
                    },
                    "execution_count": 66,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "hallucination_eval_df.head()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Observations\n",
                "\n",
                "Let's now take our results and aggregate them to get a sense of how well the LLM is answering the questions given the context."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 67,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "score    0.879781\n",
                            "dtype: float64"
                        ]
                    },
                    "execution_count": 67,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "qa_correctness_eval_df.mean(numeric_only=True)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 68,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "score    0.081967\n",
                            "dtype: float64"
                        ]
                    },
                    "execution_count": 68,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "hallucination_eval_df.mean(numeric_only=True)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Our QA Correctness score of `0.91` and a Hallucinations score `0.05` signifies that the generated answers are correct ~91% of the time and that the responses contain hallucinations 5% of the time - there is room for improvement. This could be due to the retrieval strategy or the LLM itself. We will need to investigate further to determine the root cause."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Since we have evaluated our RAG system's QA performance and Hallucinations performance, let's send these evaluations to Phoenix for visualization."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 69,
            "metadata": {},
            "outputs": [],
            "source": [
                "from phoenix.trace import SpanEvaluations\n",
                "\n",
                "px.Client().log_evaluations(\n",
                "    SpanEvaluations(dataframe=qa_correctness_eval_df, eval_name=\"Q&A Correctness\"),\n",
                "    SpanEvaluations(dataframe=hallucination_eval_df, eval_name=\"Hallucination\"),\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We now have sent all our evaluations to Phoenix. Let's go to the Phoenix application and view the results! Since we've sent all the evals to Phoenix, we can analyze the results together to make a determination on whether or not poor retrieval or irrelevant context has an effect on the LLM's ability to generate the correct response."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 70,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "phoenix URL http://localhost:6006/\n"
                    ]
                }
            ],
            "source": [
                "print(\"phoenix URL\", px.active_session().url)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Conclusion\n",
                "\n",
                "We have explored how to build and evaluate a RAG pipeline using LlamaIndex and Phoenix, with a specific focus on evaluating the retrieval system and generated responses within the pipelines. \n",
                "\n",
                "Phoenix offers a variety of other evaluations that can be used to assess the performance of your LLM Application. For more details, see the [LLM Evals](https://docs.arize.com/phoenix/llm-evals/llm-evals) documentation."
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "phoenix",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.10.3"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}
